remove the swarm wireguard mesh; tombstone its options
This commit is contained in:
atlas
parent
fbfc6f392a
commit
adc61c9043
34 changed files with 330 additions and 1068 deletions
|
|
@ -55,8 +55,7 @@ the swarm — one of each, each on whichever host you choose
|
||||||
├── swarm-grafana dashboards over both stores
|
├── swarm-grafana dashboards over both stores
|
||||||
│
|
│
|
||||||
└── opt-in, not part of singleHostSwarm
|
└── opt-in, not part of singleHostSwarm
|
||||||
├── swarm-ci-runner Forgejo Actions runner (deploy.ci-runner)
|
└── swarm-ci-runner Forgejo Actions runner (deploy.ci-runner)
|
||||||
└── wg-hive WireGuard mesh between hives (deploy.wireguard)
|
|
||||||
|
|
||||||
the hives — the substrate, as many as you like (deploy.hive-controller.enable)
|
the hives — the substrate, as many as you like (deploy.hive-controller.enable)
|
||||||
│
|
│
|
||||||
|
|
|
||||||
|
|
@ -23,10 +23,6 @@ This document contains the help content for the `hivectl` command-line program.
|
||||||
* [`hivectl agent subvol snapshot delete`↴](#hivectl-agent-subvol-snapshot-delete)
|
* [`hivectl agent subvol snapshot delete`↴](#hivectl-agent-subvol-snapshot-delete)
|
||||||
* [`hivectl agent subvol snapshot send`↴](#hivectl-agent-subvol-snapshot-send)
|
* [`hivectl agent subvol snapshot send`↴](#hivectl-agent-subvol-snapshot-send)
|
||||||
* [`hivectl list-agents`↴](#hivectl-list-agents)
|
* [`hivectl list-agents`↴](#hivectl-list-agents)
|
||||||
* [`hivectl wg`↴](#hivectl-wg)
|
|
||||||
* [`hivectl wg init`↴](#hivectl-wg-init)
|
|
||||||
* [`hivectl wg peer`↴](#hivectl-wg-peer)
|
|
||||||
* [`hivectl wg status`↴](#hivectl-wg-status)
|
|
||||||
* [`hivectl peer-config`↴](#hivectl-peer-config)
|
* [`hivectl peer-config`↴](#hivectl-peer-config)
|
||||||
* [`hivectl stop`↴](#hivectl-stop)
|
* [`hivectl stop`↴](#hivectl-stop)
|
||||||
* [`hivectl start`↴](#hivectl-start)
|
* [`hivectl start`↴](#hivectl-start)
|
||||||
|
|
@ -43,7 +39,6 @@ Sibling to the `hive-c0re` daemon binary. Covers host-side admin operations that
|
||||||
|
|
||||||
* `agent` — Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running
|
* `agent` — Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running
|
||||||
* `list-agents` — Show all managed agents with their status and technical state
|
* `list-agents` — Show all managed agents with their status and technical state
|
||||||
* `wg` — WireGuard inter-hive mesh setup helpers
|
|
||||||
* `peer-config` — Generate the federation peer-config block for THIS hive
|
* `peer-config` — Generate the federation peer-config block for THIS hive
|
||||||
* `stop` — Stop containers hive-wide in one operator action
|
* `stop` — Stop containers hive-wide in one operator action
|
||||||
* `start` — Start containers hive-wide — the inverse of `hivectl stop`
|
* `start` — Start containers hive-wide — the inverse of `hivectl stop`
|
||||||
|
|
@ -290,72 +285,13 @@ Global — not scoped to one agent, so it lives at the top level rather than und
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## `hivectl wg`
|
|
||||||
|
|
||||||
WireGuard inter-hive mesh setup helpers.
|
|
||||||
|
|
||||||
Generate this hive's mesh key and print the nix to enable the mesh, add a peer, or inspect live interface state.
|
|
||||||
|
|
||||||
**Usage:** `hivectl wg <COMMAND>`
|
|
||||||
|
|
||||||
###### **Subcommands:**
|
|
||||||
|
|
||||||
* `init` — Generate this hive's WireGuard key (if absent) and print its public key plus the nix to enable the mesh
|
|
||||||
* `peer` — Print the nix to add a peer hive to the mesh
|
|
||||||
* `status` — Show the live mesh interface state
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## `hivectl wg init`
|
|
||||||
|
|
||||||
Generate this hive's WireGuard key (if absent) and print its public key plus the nix to enable the mesh
|
|
||||||
|
|
||||||
**Usage:** `hivectl wg init [OPTIONS]`
|
|
||||||
|
|
||||||
###### **Options:**
|
|
||||||
|
|
||||||
* `--address <ADDRESS>` — This hive's mesh address (for example `10.42.0.1/32`) to bake into the printed snippet. Omit to get a placeholder you fill in
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## `hivectl wg peer`
|
|
||||||
|
|
||||||
Print the nix to add a peer hive to the mesh
|
|
||||||
|
|
||||||
**Usage:** `hivectl wg peer [OPTIONS] --pubkey <PUBKEY> --address <ADDRESS> <DOMAIN>`
|
|
||||||
|
|
||||||
###### **Arguments:**
|
|
||||||
|
|
||||||
* `<DOMAIN>` — Peer hive's DNS domain (its `swarm.hives` entry's `domain`)
|
|
||||||
|
|
||||||
###### **Options:**
|
|
||||||
|
|
||||||
* `--pubkey <PUBKEY>` — Peer's WireGuard public key (from its `hivectl wg init`)
|
|
||||||
* `--address <ADDRESS>` — Peer's mesh address (for example `10.42.0.2/32`)
|
|
||||||
* `--endpoint <ENDPOINT>` — Peer's `host:port` endpoint (omit for a peer that only dials out, for example one behind NAT — it must set an endpoint pointing back here)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## `hivectl wg status`
|
|
||||||
|
|
||||||
Show the live mesh interface state
|
|
||||||
|
|
||||||
**Usage:** `hivectl wg status`
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## `hivectl peer-config`
|
## `hivectl peer-config`
|
||||||
|
|
||||||
Generate the federation peer-config block for THIS hive.
|
Generate the federation peer-config block for THIS hive.
|
||||||
|
|
||||||
Prints the nix a peer operator pastes into their swarm config to trust and reach this hive.
|
Prints the nix a peer operator pastes into their swarm config to trust and reach this hive.
|
||||||
|
|
||||||
**Usage:** `hivectl peer-config [OPTIONS]`
|
**Usage:** `hivectl peer-config`
|
||||||
|
|
||||||
###### **Options:**
|
|
||||||
|
|
||||||
* `--wg-address <WG_ADDRESS>` — This hive's WireGuard mesh address (for example `10.42.0.1/32`), emitted as `wireguardAddress`. Omit when not running the mesh
|
|
||||||
* `--wg-endpoint <WG_ENDPOINT>` — This hive's public WireGuard endpoint (`host:port`), emitted as `wireguardEndpoint`. Omit when peers dial in / no mesh
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -126,9 +126,8 @@ if you want it. What that grant lets an agent reach on a hive is on the
|
||||||
|
|
||||||
Each hive carries an identity (`hiveName`, the `swarm.hives` directory) that
|
Each hive carries an identity (`hiveName`, the `swarm.hives` directory) that
|
||||||
says which hive it's and how to reach the others; the directory feeds the
|
says which hive it's and how to reach the others; the directory feeds the
|
||||||
swarm UI's hive list, matrix federation and the optional WireGuard mesh. →
|
swarm UI's hive list and matrix federation. →
|
||||||
[the hive directory](hive-directory.md). A private link between hives instead
|
[the hive directory](hive-directory.md).
|
||||||
of public HTTPS: [WireGuard mesh](wireguard.md).
|
|
||||||
|
|
||||||
## Cross-references
|
## Cross-references
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -95,18 +95,14 @@ mechanism.
|
||||||
swarm's root needs a CA-issued certificate (ACME). Federation firewall
|
swarm's root needs a CA-issued certificate (ACME). Federation firewall
|
||||||
and TLS requirements: [`2-handbook/swarm/matrix.md`](matrix.md).
|
and TLS requirements: [`2-handbook/swarm/matrix.md`](matrix.md).
|
||||||
|
|
||||||
3. **WireGuard mesh** (optional) — `deploy.wireguard.enable` reads each
|
|
||||||
entry's `wireguardPublicKey`/`wireguardEndpoint`/`wireguardAddress`
|
|
||||||
to configure `wg-hive`. See [the WireGuard page](wireguard.md).
|
|
||||||
|
|
||||||
## Reaching the queue and the store on a routable address
|
## Reaching the queue and the store on a routable address
|
||||||
|
|
||||||
The mesh isn't the only way to reach the queue and the secret store from
|
Other hives reach the queue and the secret store on a routable address. Set
|
||||||
another host. Set `deploy.nats.openFirewall` on the queue's host to open the
|
`deploy.nats.openFirewall` on the queue's host to open the queue's port on
|
||||||
queue's port on every interface, and `deploy.bao.openFirewall` together with
|
every interface, and `deploy.bao.openFirewall` together with
|
||||||
`deploy.bao.extraListenAddresses` on the store's host to open the store's port
|
`deploy.bao.extraListenAddresses` on the store's host to open the store's port
|
||||||
on the addresses it listens on. Both default to off. Each exposes a service
|
on the addresses it listens on. Both default to off. Each exposes a service
|
||||||
beyond the tunnel, which is the operator's decision. The queue requires TLS and
|
beyond the agent bridge, which is the operator's decision. The queue requires TLS and
|
||||||
its `auth_callout` refuses every client it can't identify. The store requires a
|
its `auth_callout` refuses every client it can't identify. The store requires a
|
||||||
client certificate on every listener, and evaluation fails when you
|
client certificate on every listener, and evaluation fails when you
|
||||||
set `deploy.bao.openFirewall` without `deploy.bao.clientCaFile`. Point
|
set `deploy.bao.openFirewall` without `deploy.bao.clientCaFile`. Point
|
||||||
|
|
|
||||||
|
|
@ -1,80 +0,0 @@
|
||||||
# WireGuard mesh between hives (optional)
|
|
||||||
|
|
||||||
The peer config in [the hive directory](hive-directory.md#hive-identity-config) uses public HTTPS for all inter-hive traffic.
|
|
||||||
For private deployments — or to reduce latency and TLS overhead on
|
|
||||||
intra-swarm traffic — hyperhive can configure a host-to-host
|
|
||||||
WireGuard mesh.
|
|
||||||
|
|
||||||
## Generating keys
|
|
||||||
|
|
||||||
On each hive host:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
|
|
||||||
wg pubkey < /etc/wireguard/hive.key # → share this with peer operators
|
|
||||||
```
|
|
||||||
|
|
||||||
## Config example (two hives)
|
|
||||||
|
|
||||||
```nix
|
|
||||||
# hive A (firsthive.myswarm.example, mesh IP 10.100.0.1)
|
|
||||||
services.hyperhive = {
|
|
||||||
deploy.wireguard = {
|
|
||||||
enable = true;
|
|
||||||
privateKeyFile = "/etc/wireguard/hive.key";
|
|
||||||
address = "10.100.0.1/24";
|
|
||||||
listenPort = 51820; # optional, default 51820
|
|
||||||
};
|
|
||||||
|
|
||||||
# The same `hives` attrset both hosts hold — mesh fields included,
|
|
||||||
# since "where this hive can be dialled" is a fact about that hive.
|
|
||||||
swarm.hives = {
|
|
||||||
firsthive = {
|
|
||||||
domain = "firsthive.myswarm.example";
|
|
||||||
wireguardPublicKey = "base64keyA=";
|
|
||||||
wireguardEndpoint = "198.51.100.1:51820";
|
|
||||||
wireguardAddress = "10.100.0.1/32";
|
|
||||||
};
|
|
||||||
secondhive = {
|
|
||||||
domain = "secondhive.elsewhere.example";
|
|
||||||
wireguardPublicKey = "base64keyB=";
|
|
||||||
wireguardEndpoint = "203.0.113.42:51820";
|
|
||||||
wireguardAddress = "10.100.0.2/32";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# hive B (secondhive.elsewhere.example, mesh IP 10.100.0.2)
|
|
||||||
services.hyperhive = {
|
|
||||||
deploy.wireguard = {
|
|
||||||
enable = true;
|
|
||||||
privateKeyFile = "/etc/wireguard/hive.key";
|
|
||||||
address = "10.100.0.2/24";
|
|
||||||
};
|
|
||||||
|
|
||||||
swarm.hives = {
|
|
||||||
# … identical to hive A's …
|
|
||||||
};
|
|
||||||
};
|
|
||||||
```
|
|
||||||
|
|
||||||
## What the mesh does
|
|
||||||
|
|
||||||
- hyperhive configures `networking.wireguard.interfaces.wg-hive` on the
|
|
||||||
host (not inside agent containers; containers reach peers via the
|
|
||||||
host's routing table).
|
|
||||||
- It opens UDP port 51820 (or `listenPort`) on the host firewall.
|
|
||||||
- It reads each entry's `wireguardAddress` from `services.hyperhive.swarm.hives`,
|
|
||||||
minus this host's own hive, to build `wg-hive`'s `allowedIPs`, so intra-swarm
|
|
||||||
traffic can route over the mesh address rather than the public domain.
|
|
||||||
- It sets `persistentKeepalive = 25` by default; override or null to
|
|
||||||
disable (not needed when both sides have public IPs and no NAT).
|
|
||||||
|
|
||||||
## NAT / one-sided endpoints
|
|
||||||
|
|
||||||
If one host is behind NAT and can't accept incoming connections, only
|
|
||||||
that host needs a null `wireguardEndpoint` on the peer config — the
|
|
||||||
other side initiates. With keepalive on, the NAT hole stays open.
|
|
||||||
|
|
||||||
If both hosts are behind NAT, you need a STUN relay or a third host
|
|
||||||
(exit node); hyperhive sets up neither.
|
|
||||||
|
|
@ -29,7 +29,6 @@ that touches it.
|
||||||
│ squid 10.42.0.1:3128 (egress proxy) [hive-squid] │
|
│ squid 10.42.0.1:3128 (egress proxy) [hive-squid] │
|
||||||
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
|
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
|
||||||
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
|
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
|
||||||
│ wg-hive :51820/udp — swarm mesh, when enabled (host iface) │
|
|
||||||
│ │
|
│ │
|
||||||
│ hive-br0 10.42.0.1/24 │
|
│ hive-br0 10.42.0.1/24 │
|
||||||
│ ┌──────────┼──────────────┐ │
|
│ ┌──────────┼──────────────┐ │
|
||||||
|
|
@ -72,8 +71,8 @@ The flows, end to end:
|
||||||
UI unix socket.
|
UI unix socket.
|
||||||
- **Internet egress** — agent default route points at the bridge IP;
|
- **Internet egress** — agent default route points at the bridge IP;
|
||||||
the host forwards + masquerades out its uplink.
|
the host forwards + masquerades out its uplink.
|
||||||
- **Swarm** — peer hives connect over the `wg-hive` WireGuard mesh
|
- **Swarm** — peer hives reach each other's gateway and forge over HTTPS, and
|
||||||
and reach each other's gateway/forge across it
|
the queue and the secret store on their routable listeners
|
||||||
([`docs/2-handbook/swarm/`](../../2-handbook/swarm/README.md)).
|
([`docs/2-handbook/swarm/`](../../2-handbook/swarm/README.md)).
|
||||||
- **Control plane (no network)** — per-agent broker socket
|
- **Control plane (no network)** — per-agent broker socket
|
||||||
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
|
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
|
||||||
|
|
@ -526,7 +525,7 @@ squid (`services.hyperhive.deploy.hive-controller.egress.intercept.enable`).
|
||||||
|
|
||||||
- **Who gets redirected.** Every flow from the bridge to port 80 or 443, except
|
- **Who gets redirected.** Every flow from the bridge to port 80 or 443, except
|
||||||
a flow to an address that stays on the bridge, to one of the host's own
|
a flow to an address that stays on the bridge, to one of the host's own
|
||||||
addresses or to the `wg-hive` interface, and a flow from a MAC in the set
|
addresses, and a flow from a MAC in the set
|
||||||
`direct_mac`. That set holds the MACs of the ports of every container that's
|
`direct_mac`. That set holds the MACs of the ports of every container that's
|
||||||
not an agent (the swarm's service containers) and of the agents listed in
|
not an agent (the swarm's service containers) and of the agents listed in
|
||||||
`services.hyperhive.deploy.hive-controller.squid.unrestrictedFile`. So
|
`services.hyperhive.deploy.hive-controller.squid.unrestrictedFile`. So
|
||||||
|
|
@ -727,13 +726,13 @@ loopback — the hive-c0re admin socket is a UDS, not TCP.
|
||||||
|
|
||||||
`services.hyperhive.deploy.hive-controller.egress.forward` sets what the `forward` chain of
|
`services.hyperhive.deploy.hive-controller.egress.forward` sets what the `forward` chain of
|
||||||
the `inet hive_egress` table does with traffic the host routes from the agent
|
the `inet hive_egress` table does with traffic the host routes from the agent
|
||||||
bridge to anywhere but the bridge and the `wg-hive` mesh interface. It takes
|
bridge to anywhere but the bridge. It takes
|
||||||
`accept`, `log` (the default) or `reject`. In `log` mode the chain blocks
|
`accept`, `log` (the default) or `reject`. In `log` mode the chain blocks
|
||||||
nothing. It logs each new flow and counts every packet. In `reject` mode it
|
nothing. It logs each new flow and counts every packet. In `reject` mode it
|
||||||
also refuses the flow.
|
also refuses the flow.
|
||||||
|
|
||||||
- **Who is exempt.** In `log` and `reject` mode alike, the chain accepts the
|
- **Who is exempt.** In `log` and `reject` mode alike, the chain accepts the
|
||||||
mesh traffic first, then the traffic of every MAC in the `direct_mac` set: the
|
traffic of every MAC in the `direct_mac` set: the
|
||||||
`unrestricted` agents and the service containers, which the `intercept` chain
|
`unrestricted` agents and the service containers, which the `intercept` chain
|
||||||
leaves alone too (see _Intercepted web traffic_). They leave no line, so a
|
leaves alone too (see _Intercepted web traffic_). They leave no line, so a
|
||||||
`hive-egress-would-drop` line is exactly a flow that `reject` would refuse,
|
`hive-egress-would-drop` line is exactly a flow that `reject` would refuse,
|
||||||
|
|
|
||||||
|
|
@ -128,9 +128,7 @@ by hand, start the workflow from the forge UI (Actions, vm-nightly, Run
|
||||||
workflow, optional `test` input), or run
|
workflow, optional `test` input), or run
|
||||||
`nix build .#vmTests.x86_64-linux.<name>` on a host with KVM. Current tests:
|
`nix build .#vmTests.x86_64-linux.<name>` on a host with KVM. Current tests:
|
||||||
`agent-ephemeral-rootfs` (the switch of an agent container to an ephemeral
|
`agent-ephemeral-rootfs` (the switch of an agent container to an ephemeral
|
||||||
root, through the real hive-priv unit), `hive-squid`, and `wg-hive` (two hives
|
root, through the real hive-priv unit) and `hive-squid`.
|
||||||
peered over the `wg-hive` mesh: handshake, traffic, and a wrong-key peer that
|
|
||||||
must not handshake).
|
|
||||||
|
|
||||||
The `full-setup` script is plain Python in `nix/vm-tests/full-setup.py`, with no
|
The `full-setup` script is plain Python in `nix/vm-tests/full-setup.py`, with no
|
||||||
Nix interpolation. Commands the script runs in the VM find their binaries on the
|
Nix interpolation. Commands the script runs in the VM find their binaries on the
|
||||||
|
|
|
||||||
|
|
@ -72,9 +72,9 @@ The queue URL is the same string on every hive. The queue accepts TLS only,
|
||||||
with a certificate for `swarm.nats.domain` (default `nats.<swarm.domain>`) and
|
with a certificate for `swarm.nats.domain` (default `nats.<swarm.domain>`) and
|
||||||
no other name or address, so a URL with an IP address or `nats://` fails.
|
no other name or address, so a URL with an IP address or `nats://` fails.
|
||||||
The queue's host resolves the name itself. **A multi-host swarm needs one
|
The queue's host resolves the name itself. **A multi-host swarm needs one
|
||||||
upstream DNS record**, `nats.<swarm.domain>` pointing at the queue host's mesh
|
upstream DNS record**, `nats.<swarm.domain>` pointing at the queue host's routable
|
||||||
address, the same contract as `bao.<swarm.domain>`. The queue's port is open
|
address, the same contract as `bao.<swarm.domain>`. The queue's port is open
|
||||||
on `wg-hive` when that host is on the mesh.
|
host-wide only with `deploy.nats.openFirewall`.
|
||||||
|
|
||||||
The queue's host asks the secret store for that certificate under a store
|
The queue's host asks the secret store for that certificate under a store
|
||||||
leaf of its own, `deploy.nats.baoClientCertFile` / `baoClientKeyFile`. On the
|
leaf of its own, `deploy.nats.baoClientCertFile` / `baoClientKeyFile`. On the
|
||||||
|
|
|
||||||
|
|
@ -148,7 +148,6 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](../../2-hand
|
||||||
| the CI run poller's copy of its OIDC secret | authelia's mint, which creates it in the swarm secret store | `swarm-ci-metrics` reads `swarm/services/<swarm.ci-metrics.clientId>/oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else |
|
| the CI run poller's copy of its OIDC secret | authelia's mint, which creates it in the swarm secret store | `swarm-ci-metrics` reads `swarm/services/<swarm.ci-metrics.clientId>/oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else |
|
||||||
| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's secret (`swarm.authelia.otel.clientId`). It creates it at `swarm/services/<id>/oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf |
|
| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's secret (`swarm.authelia.otel.clientId`). It creates it at `swarm/services/<id>/oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf |
|
||||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||||
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
|
||||||
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||||
| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||||
| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place |
|
| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place |
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@
|
||||||
The operator-facing host CLI. A thin client for the `hive-c0re`
|
The operator-facing host CLI. A thin client for the `hive-c0re`
|
||||||
daemon — speaks the host admin socket protocol (`hive-host-sock`) and
|
daemon — speaks the host admin socket protocol (`hive-host-sock`) and
|
||||||
does not link the daemon crate. Container lifecycle and the
|
does not link the daemon crate. Container lifecycle and the
|
||||||
gateway verbs all forward to the daemon and need it running; a few (`wg`/`peer-config`, `choom`) work off local
|
gateway verbs all forward to the daemon and need it running; a few (`peer-config`, `choom`) work off local
|
||||||
host state instead.
|
host state instead.
|
||||||
|
|
||||||
## When to use it
|
## When to use it
|
||||||
|
|
@ -25,7 +25,7 @@ dispatch:
|
||||||
- **`power.rs`** — restart/start/stop at the container level.
|
- **`power.rs`** — restart/start/stop at the container level.
|
||||||
- **`choom.rs`** — drop into an interactive claude session in a
|
- **`choom.rs`** — drop into an interactive claude session in a
|
||||||
container.
|
container.
|
||||||
- **`wg.rs`** — WireGuard mesh helpers.
|
- **`peer_config.rs`** — the federation block a peer pastes.
|
||||||
- **`subvol.rs`** — btrfs state-subvolume ops.
|
- **`subvol.rs`** — btrfs state-subvolume ops.
|
||||||
- **`quota.rs`**, **`util.rs`**, **`completions.rs`** — shared helpers
|
- **`quota.rs`**, **`util.rs`**, **`completions.rs`** — shared helpers
|
||||||
and shell completions.
|
and shell completions.
|
||||||
|
|
|
||||||
|
|
@ -51,28 +51,11 @@ pub enum Cmd {
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
json: bool,
|
json: bool,
|
||||||
},
|
},
|
||||||
/// WireGuard inter-hive mesh setup helpers.
|
|
||||||
///
|
|
||||||
/// Generate this hive's mesh key and print the nix to enable the mesh,
|
|
||||||
/// add a peer, or inspect live interface state.
|
|
||||||
Wg {
|
|
||||||
#[command(subcommand)]
|
|
||||||
cmd: WgCmd,
|
|
||||||
},
|
|
||||||
/// Generate the federation peer-config block for THIS hive.
|
/// Generate the federation peer-config block for THIS hive.
|
||||||
///
|
///
|
||||||
/// Prints the nix a peer operator pastes into their swarm config to
|
/// Prints the nix a peer operator pastes into their swarm config to
|
||||||
/// trust and reach this hive.
|
/// trust and reach this hive.
|
||||||
PeerConfig {
|
PeerConfig,
|
||||||
/// This hive's WireGuard mesh address (for example `10.42.0.1/32`),
|
|
||||||
/// emitted as `wireguardAddress`. Omit when not running the mesh.
|
|
||||||
#[arg(long)]
|
|
||||||
wg_address: Option<String>,
|
|
||||||
/// This hive's public WireGuard endpoint (`host:port`), emitted as
|
|
||||||
/// `wireguardEndpoint`. Omit when peers dial in / no mesh.
|
|
||||||
#[arg(long)]
|
|
||||||
wg_endpoint: Option<String>,
|
|
||||||
},
|
|
||||||
/// Stop containers hive-wide in one operator action.
|
/// Stop containers hive-wide in one operator action.
|
||||||
///
|
///
|
||||||
/// Bare `hivectl stop` stops every agent and the gateway; scope flags
|
/// Bare `hivectl stop` stops every agent and the gateway; scope flags
|
||||||
|
|
@ -164,35 +147,6 @@ impl ScopeArgs {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Subcommand)]
|
|
||||||
pub enum WgCmd {
|
|
||||||
/// Generate this hive's WireGuard key (if absent) and print its public
|
|
||||||
/// key plus the nix to enable the mesh.
|
|
||||||
Init {
|
|
||||||
/// This hive's mesh address (for example `10.42.0.1/32`) to bake into the
|
|
||||||
/// printed snippet. Omit to get a placeholder you fill in.
|
|
||||||
#[arg(long)]
|
|
||||||
address: Option<String>,
|
|
||||||
},
|
|
||||||
/// Print the nix to add a peer hive to the mesh.
|
|
||||||
Peer {
|
|
||||||
/// Peer hive's DNS domain (its `swarm.hives` entry's `domain`)
|
|
||||||
domain: String,
|
|
||||||
/// Peer's WireGuard public key (from its `hivectl wg init`).
|
|
||||||
#[arg(long)]
|
|
||||||
pubkey: String,
|
|
||||||
/// Peer's mesh address (for example `10.42.0.2/32`).
|
|
||||||
#[arg(long)]
|
|
||||||
address: String,
|
|
||||||
/// Peer's `host:port` endpoint (omit for a peer that only dials out,
|
|
||||||
/// for example one behind NAT — it must set an endpoint pointing back here).
|
|
||||||
#[arg(long)]
|
|
||||||
endpoint: Option<String>,
|
|
||||||
},
|
|
||||||
/// Show the live mesh interface state.
|
|
||||||
Status,
|
|
||||||
}
|
|
||||||
|
|
||||||
// Default host admin socket path. Shared with `hive-c0re`'s `main.rs`
|
// Default host admin socket path. Shared with `hive-c0re`'s `main.rs`
|
||||||
// default via `hive_host_sock::HOST_SOCKET` — the daemon binds there
|
// default via `hive_host_sock::HOST_SOCKET` — the daemon binds there
|
||||||
// and `hivectl` connects to it.
|
// and `hivectl` connects to it.
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,7 @@
|
||||||
//! `gateway`) all forward to the daemon, which owns the broker,
|
//! `gateway`) all forward to the daemon, which owns the broker,
|
||||||
//! the credentials, and the provisioning logic — a running daemon is
|
//! the credentials, and the provisioning logic — a running daemon is
|
||||||
//! required for those. A couple of verbs work off local host state
|
//! required for those. A couple of verbs work off local host state
|
||||||
//! directly instead (`wg` / `peer-config` read the mesh key + TLS CA), so
|
//! directly instead (`peer-config` reads the TLS CA), so
|
||||||
//! they don't need the socket. `agent <name> choom` execs into a container
|
//! they don't need the socket. `agent <name> choom` execs into a container
|
||||||
//! rather than asking the daemon to do anything, but still uses the socket
|
//! rather than asking the daemon to do anything, but still uses the socket
|
||||||
//! for its "is this an agent?" pre-flight — that answer lives in a
|
//! for its "is this an agent?" pre-flight — that answer lives in a
|
||||||
|
|
@ -26,12 +26,12 @@ mod client;
|
||||||
/// Rebuild-queue node progress rendering (`wait_for_nodes` + the spinner /
|
/// Rebuild-queue node progress rendering (`wait_for_nodes` + the spinner /
|
||||||
/// plain renderers), split out to keep this file manageable.
|
/// plain renderers), split out to keep this file manageable.
|
||||||
mod dag_progress;
|
mod dag_progress;
|
||||||
use cli::{Cli, Cmd, WgCmd};
|
use cli::{Cli, Cmd};
|
||||||
mod completions;
|
mod completions;
|
||||||
mod util;
|
mod util;
|
||||||
use completions::generate_completions;
|
use completions::generate_completions;
|
||||||
mod wg;
|
mod peer_config;
|
||||||
use wg::{peer_config, require_hive_domain, wg_init, wg_peer, wg_status};
|
use peer_config::{peer_config, require_hive_domain};
|
||||||
mod agents;
|
mod agents;
|
||||||
mod choom;
|
mod choom;
|
||||||
mod watch;
|
mod watch;
|
||||||
|
|
@ -53,25 +53,9 @@ async fn main() -> Result<()> {
|
||||||
match cli.cmd {
|
match cli.cmd {
|
||||||
Cmd::Agent { name, cmd } => run_agent(&socket, &name, cmd).await,
|
Cmd::Agent { name, cmd } => run_agent(&socket, &name, cmd).await,
|
||||||
Cmd::ListAgents { json } => agents_list(&socket, json).await,
|
Cmd::ListAgents { json } => agents_list(&socket, json).await,
|
||||||
Cmd::Wg { cmd } => match cmd {
|
Cmd::PeerConfig => {
|
||||||
WgCmd::Init { address } => wg_init(&socket, address.as_deref()).await,
|
|
||||||
WgCmd::Peer {
|
|
||||||
domain,
|
|
||||||
pubkey,
|
|
||||||
address,
|
|
||||||
endpoint,
|
|
||||||
} => {
|
|
||||||
wg_peer(&domain, &pubkey, &address, endpoint.as_deref());
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
WgCmd::Status => wg_status(),
|
|
||||||
},
|
|
||||||
Cmd::PeerConfig {
|
|
||||||
wg_address,
|
|
||||||
wg_endpoint,
|
|
||||||
} => {
|
|
||||||
let domain = require_hive_domain(&socket).await?;
|
let domain = require_hive_domain(&socket).await?;
|
||||||
peer_config(&domain, wg_address.as_deref(), wg_endpoint.as_deref());
|
peer_config(&domain);
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
Cmd::Stop {
|
Cmd::Stop {
|
||||||
|
|
|
||||||
143
hivectl/src/peer_config.rs
Normal file
143
hivectl/src/peer_config.rs
Normal file
|
|
@ -0,0 +1,143 @@
|
||||||
|
//! `hivectl peer-config` — print the block a peer operator pastes to
|
||||||
|
//! federate with this hive.
|
||||||
|
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use anyhow::{Context as _, Result};
|
||||||
|
|
||||||
|
use crate::util::query_hive_urls;
|
||||||
|
|
||||||
|
/// Host path of this hive's TLS trust bundle (matches the
|
||||||
|
/// `services.hyperhive.deploy.hive-controller.tls.stateDir` default in
|
||||||
|
/// hive-tls.nix). Its existence means the gateway is in self-signed mode,
|
||||||
|
/// serving a leaf the secret store issued, so a federating peer needs an
|
||||||
|
/// anchor for it. Absent = ACME / operator cert, trusted by the default CA
|
||||||
|
/// bundle with nothing to distribute.
|
||||||
|
const HIVE_TLS_TRUST_BUNDLE_PATH: &str = "/var/lib/hive-tls/trust-bundle.pem";
|
||||||
|
|
||||||
|
/// Host path of the secret store's `pki` root (hive-tls.nix's
|
||||||
|
/// `$svcroot`, folded into the trust bundle). The anchor a whole swarm
|
||||||
|
/// shares: every hive's gateway leaf and the swarm's service names are
|
||||||
|
/// issued under it, so a peer host installs it once and every present
|
||||||
|
/// *and future* hive validates.
|
||||||
|
const STORE_PKI_ROOT_PATH: &str = "/var/lib/hive-tls/swarm-services-root.pem";
|
||||||
|
|
||||||
|
/// The option whose value keys a hive's entry in `services.hyperhive.swarm.hives`.
|
||||||
|
const HIVE_NAME_OPTION: &str = "services.hyperhive.deploy.hive-controller.hiveName";
|
||||||
|
|
||||||
|
/// Attrset key for a hive in `services.hyperhive.swarm.hives`, derived
|
||||||
|
/// from its domain's first DNS label.
|
||||||
|
///
|
||||||
|
/// A hive occupies `<hiveName>.<swarm.domain>`, so the first label *is*
|
||||||
|
/// the hive name in any deployment that hasn't overridden `domain` by
|
||||||
|
/// hand. Where it has, a wrong key fails loudly rather than quietly: on
|
||||||
|
/// that hive's own host the `hives.<hiveName>` assertion fires, because
|
||||||
|
/// the directory is supposed to be the same attrset everywhere. The
|
||||||
|
/// snippet below says so rather than presenting the guess as fact.
|
||||||
|
fn hive_key(domain: &str) -> &str {
|
||||||
|
domain.split('.').next().unwrap_or(domain)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Require this hive's domain from the daemon for snippet generation.
|
||||||
|
/// Errors with a clear hint when it can't be resolved, so `peer-config`
|
||||||
|
/// never silently emits a wrong key. An unreachable daemon propagates its
|
||||||
|
/// own (classified) connect error; only a *reachable* daemon with no domain
|
||||||
|
/// gets the config hint.
|
||||||
|
pub(crate) async fn require_hive_domain(socket: &Path) -> Result<String> {
|
||||||
|
query_hive_urls(socket)
|
||||||
|
.await
|
||||||
|
.context("could not determine this hive's domain from the daemon")?
|
||||||
|
.and_then(|u| u.domain)
|
||||||
|
.context("the daemon reported no domain — set `services.hyperhive.domain`")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The first closing comment line of the `peer-config` snippet.
|
||||||
|
fn peer_config_key_note() -> String {
|
||||||
|
format!("# the key must be this hive's {HIVE_NAME_OPTION}, and the")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The `peer-config` step carrying the secret store's `pki` root: the
|
||||||
|
/// certificate itself, or where it will appear when this hive has not
|
||||||
|
/// fetched it yet.
|
||||||
|
fn store_root_step(pem: Option<&str>) -> String {
|
||||||
|
const HEAD: &str = "# 1. trust the SECRET STORE's pki root on the peer host (once, not per hive);\n\
|
||||||
|
# every hive's gateway leaf and the swarm's service names are issued under it:\n";
|
||||||
|
match pem.map(str::trim_end).filter(|p| !p.is_empty()) {
|
||||||
|
Some(p) => format!(
|
||||||
|
"{HEAD}{p}\n\
|
||||||
|
# (a self-signed hive in this swarm fetches it itself, into {STORE_PKI_ROOT_PATH})\n"
|
||||||
|
),
|
||||||
|
None => format!(
|
||||||
|
"{HEAD}# not on this host yet: swarm-services-cert.service writes it to\n\
|
||||||
|
# {STORE_PKI_ROOT_PATH}; rerun once that unit has succeeded\n"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `peer-config` — print the `swarm.hives."<name>"` block a peer
|
||||||
|
/// operator pastes to federate with THIS hive, plus the store-root step
|
||||||
|
/// when this hive serves a self-signed chain. Reads local state only (the
|
||||||
|
/// TLS trust bundle and the store root); prints, never mutates.
|
||||||
|
pub(crate) fn peer_config(domain: &str) {
|
||||||
|
let self_signed = Path::new(HIVE_TLS_TRUST_BUNDLE_PATH).exists();
|
||||||
|
let key = hive_key(domain);
|
||||||
|
|
||||||
|
if self_signed {
|
||||||
|
let store_root = std::fs::read_to_string(STORE_PKI_ROOT_PATH).ok();
|
||||||
|
print!("{}", store_root_step(store_root.as_deref()));
|
||||||
|
println!();
|
||||||
|
println!("# 2. paste into the peer hive's NixOS config:");
|
||||||
|
} else {
|
||||||
|
println!("# paste into the peer hive's NixOS config:");
|
||||||
|
}
|
||||||
|
println!("services.hyperhive.swarm.hives.\"{key}\" = {{");
|
||||||
|
println!(" domain = \"{domain}\";");
|
||||||
|
println!("}};");
|
||||||
|
println!("{}", peer_config_key_note());
|
||||||
|
println!("# same entry belongs in every hive's config — `hives` is the swarm.");
|
||||||
|
if !self_signed {
|
||||||
|
println!(
|
||||||
|
"# (this hive's cert chains to a public CA — nothing to install; \
|
||||||
|
it's trusted by the default bundle.)"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::{STORE_PKI_ROOT_PATH, peer_config_key_note, store_root_step};
|
||||||
|
|
||||||
|
const PEM: &str = "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----";
|
||||||
|
|
||||||
|
/// The snippet tells the operator to set the deploy option, never the
|
||||||
|
/// removed `services.hyperhive` spelling.
|
||||||
|
#[test]
|
||||||
|
fn key_note_names_the_deploy_option() {
|
||||||
|
let note = peer_config_key_note();
|
||||||
|
assert!(
|
||||||
|
note.contains("services.hyperhive.deploy.hive-controller.hiveName"),
|
||||||
|
"{note}"
|
||||||
|
);
|
||||||
|
assert!(!note.contains("services.hyperhive.hiveName"), "{note}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The certificate is printed whole, on lines of its own, so a peer
|
||||||
|
/// operator can copy it out of the output as a PEM file.
|
||||||
|
#[test]
|
||||||
|
fn store_root_is_printed_verbatim() {
|
||||||
|
let s = store_root_step(Some(&format!("{PEM}\n")));
|
||||||
|
assert!(s.contains(&format!("\n{PEM}\n")), "{s}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A hive that has not fetched the root yet says where it will be,
|
||||||
|
/// rather than printing an empty step.
|
||||||
|
#[test]
|
||||||
|
fn missing_store_root_names_the_path_and_the_unit() {
|
||||||
|
for pem in [None, Some(""), Some("\n")] {
|
||||||
|
let s = store_root_step(pem);
|
||||||
|
assert!(s.contains(STORE_PKI_ROOT_PATH), "{s}");
|
||||||
|
assert!(s.contains("swarm-services-cert.service"), "{s}");
|
||||||
|
assert!(!s.contains("BEGIN CERTIFICATE"), "{s}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -1,365 +0,0 @@
|
||||||
//! `hivectl wg` / `peer-config` — WireGuard mesh helpers: generate this
|
|
||||||
//! hive's key + the nix snippets to enable the mesh, add a peer, print the
|
|
||||||
//! block a peer pastes to federate with us, and show the live interface.
|
|
||||||
|
|
||||||
use std::path::Path;
|
|
||||||
|
|
||||||
use anyhow::{Context as _, Result, bail};
|
|
||||||
|
|
||||||
use crate::util::query_hive_urls;
|
|
||||||
|
|
||||||
/// Host path of this hive's WireGuard private key (matches the
|
|
||||||
/// `privateKeyFile` example in the deploy.wireguard nix options).
|
|
||||||
const WG_KEY_PATH: &str = "/etc/wireguard/hive.key";
|
|
||||||
/// The mesh interface name hive-c0re's nix module brings up.
|
|
||||||
const WG_INTERFACE: &str = "wg-hive";
|
|
||||||
|
|
||||||
/// Host path of this hive's TLS trust bundle (matches the
|
|
||||||
/// `services.hyperhive.deploy.hive-controller.tls.stateDir` default in
|
|
||||||
/// hive-tls.nix). Its existence means the gateway is in self-signed mode,
|
|
||||||
/// serving a leaf the secret store issued, so a federating peer needs an
|
|
||||||
/// anchor for it. Absent = ACME / operator cert, trusted by the default CA
|
|
||||||
/// bundle with nothing to distribute.
|
|
||||||
const HIVE_TLS_TRUST_BUNDLE_PATH: &str = "/var/lib/hive-tls/trust-bundle.pem";
|
|
||||||
|
|
||||||
/// Host path of the secret store's `pki` root (hive-tls.nix's
|
|
||||||
/// `$svcroot`, folded into the trust bundle). The anchor a whole swarm
|
|
||||||
/// shares: every hive's gateway leaf and the swarm's service names are
|
|
||||||
/// issued under it, so a peer host installs it once and every present
|
|
||||||
/// *and future* hive validates.
|
|
||||||
const STORE_PKI_ROOT_PATH: &str = "/var/lib/hive-tls/swarm-services-root.pem";
|
|
||||||
|
|
||||||
/// The option whose value keys a hive's entry in `services.hyperhive.swarm.hives`.
|
|
||||||
const HIVE_NAME_OPTION: &str = "services.hyperhive.deploy.hive-controller.hiveName";
|
|
||||||
|
|
||||||
/// Attrset key for a hive in `services.hyperhive.swarm.hives`, derived
|
|
||||||
/// from its domain's first DNS label.
|
|
||||||
///
|
|
||||||
/// A hive occupies `<hiveName>.<swarm.domain>`, so the first label *is*
|
|
||||||
/// the hive name in any deployment that hasn't overridden `domain` by
|
|
||||||
/// hand. Where it has, a wrong key fails loudly rather than quietly: on
|
|
||||||
/// that hive's own host the `hives.<hiveName>` assertion fires, because
|
|
||||||
/// the directory is supposed to be the same attrset everywhere. The
|
|
||||||
/// snippets below say so rather than presenting the guess as fact.
|
|
||||||
fn hive_key(domain: &str) -> &str {
|
|
||||||
domain.split('.').next().unwrap_or(domain)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Best-effort query for this hive's domain from the running daemon
|
|
||||||
/// (`HostRequest::Urls`, which reads `HYPERHIVE_HIVE_DOMAIN` from c0re's
|
|
||||||
/// service env). `None` when the daemon is unreachable or the domain is
|
|
||||||
/// unset — for the callers that treat both as "skip the optional block".
|
|
||||||
/// Use [`require_hive_domain`] where the distinction matters: it keeps the
|
|
||||||
/// connect error (and its hint) instead of flattening it away.
|
|
||||||
async fn query_hive_domain(socket: &Path) -> Option<String> {
|
|
||||||
query_hive_urls(socket).await.ok().flatten()?.domain
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Require this hive's domain from the daemon for snippet generation.
|
|
||||||
/// Errors with a clear hint when it can't be resolved, so `peer-config`
|
|
||||||
/// never silently emits a wrong key. An unreachable daemon propagates its
|
|
||||||
/// own (classified) connect error; only a *reachable* daemon with no domain
|
|
||||||
/// gets the config hint.
|
|
||||||
pub(crate) async fn require_hive_domain(socket: &Path) -> Result<String> {
|
|
||||||
query_hive_urls(socket)
|
|
||||||
.await
|
|
||||||
.context("could not determine this hive's domain from the daemon")?
|
|
||||||
.and_then(|u| u.domain)
|
|
||||||
.context("the daemon reported no domain — set `services.hyperhive.domain`")
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `wg init` — generate (if absent) the hive's WireGuard key, print its
|
|
||||||
/// public key + the nix snippet to enable the mesh, then (best-effort)
|
|
||||||
/// the `peer-config` block peers paste to federate with this hive, so a
|
|
||||||
/// fresh setup is one command. The domain comes from the daemon; if it
|
|
||||||
/// can't be resolved, the peer block is skipped (init still succeeds —
|
|
||||||
/// its core job is enabling the mesh locally).
|
|
||||||
pub(crate) async fn wg_init(socket: &Path, address: Option<&str>) -> Result<()> {
|
|
||||||
use std::os::unix::fs::PermissionsExt as _;
|
|
||||||
let key_path = Path::new(WG_KEY_PATH);
|
|
||||||
if key_path.exists() {
|
|
||||||
println!(
|
|
||||||
"WireGuard private key already present at {WG_KEY_PATH} — reusing (not regenerating)."
|
|
||||||
);
|
|
||||||
} else {
|
|
||||||
if let Some(parent) = key_path.parent() {
|
|
||||||
std::fs::create_dir_all(parent)
|
|
||||||
.with_context(|| format!("create {}", parent.display()))?;
|
|
||||||
std::fs::set_permissions(parent, std::fs::Permissions::from_mode(0o700)).ok();
|
|
||||||
}
|
|
||||||
let out = std::process::Command::new("wg")
|
|
||||||
.arg("genkey")
|
|
||||||
.output()
|
|
||||||
.context("run `wg genkey` (is wireguard-tools installed?)")?;
|
|
||||||
if !out.status.success() {
|
|
||||||
bail!(
|
|
||||||
"wg genkey failed: {}",
|
|
||||||
String::from_utf8_lossy(&out.stderr).trim()
|
|
||||||
);
|
|
||||||
}
|
|
||||||
write_key_0400(key_path, &out.stdout)?;
|
|
||||||
println!("Generated WireGuard private key at {WG_KEY_PATH} (0400).");
|
|
||||||
}
|
|
||||||
|
|
||||||
let privkey =
|
|
||||||
std::fs::read(key_path).with_context(|| format!("read {}", key_path.display()))?;
|
|
||||||
let pubkey = wg_pubkey(&privkey)?;
|
|
||||||
let addr = address.unwrap_or("<MESH_ADDRESS e.g. 10.42.0.1/32>");
|
|
||||||
println!("\nPublic key (share this with peer hives — they pass it to `hivectl wg peer`):");
|
|
||||||
println!(" {pubkey}");
|
|
||||||
println!("\nAdd to this hive's NixOS config:");
|
|
||||||
println!(" services.hyperhive.deploy.wireguard = {{");
|
|
||||||
println!(" enable = true;");
|
|
||||||
println!(" privateKeyFile = \"{WG_KEY_PATH}\";");
|
|
||||||
println!(" address = \"{addr}\";");
|
|
||||||
println!(" # listenPort = 51820; # default");
|
|
||||||
println!(" }};");
|
|
||||||
|
|
||||||
// Also print the block a peer pastes to federate with us (CA + this
|
|
||||||
// mesh key) — one-stop setup. Domain comes from the daemon;
|
|
||||||
// best-effort, so init still succeeds when it can't be resolved.
|
|
||||||
if let Some(d) = query_hive_domain(socket).await {
|
|
||||||
println!();
|
|
||||||
peer_config(&d, address, None);
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Write `key` to a sibling temp file created `0400` and rename it over
|
|
||||||
/// `path`, so the key is never readable by anyone else and a crash never
|
|
||||||
/// leaves half of it at `path`. Errors carry paths, never the key.
|
|
||||||
fn write_key_0400(path: &Path, key: &[u8]) -> Result<()> {
|
|
||||||
use std::io::Write as _;
|
|
||||||
use std::os::unix::fs::OpenOptionsExt as _;
|
|
||||||
|
|
||||||
let mut tmp_name = path
|
|
||||||
.file_name()
|
|
||||||
.with_context(|| format!("{} has no file name", path.display()))?
|
|
||||||
.to_owned();
|
|
||||||
tmp_name.push(".tmp");
|
|
||||||
let tmp = path.with_file_name(tmp_name);
|
|
||||||
// A temp file left by a crashed earlier write would fail `create_new`.
|
|
||||||
let _ = std::fs::remove_file(&tmp);
|
|
||||||
let mut f = std::fs::OpenOptions::new()
|
|
||||||
.write(true)
|
|
||||||
.create_new(true)
|
|
||||||
.mode(0o400)
|
|
||||||
.open(&tmp)
|
|
||||||
.with_context(|| format!("create {}", tmp.display()))?;
|
|
||||||
f.write_all(key)
|
|
||||||
.and_then(|()| f.sync_all())
|
|
||||||
.with_context(|| format!("write {}", tmp.display()))?;
|
|
||||||
std::fs::rename(&tmp, path).with_context(|| format!("rename onto {}", path.display()))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Derive a WireGuard public key from a private key by piping it through
|
|
||||||
/// `wg pubkey`.
|
|
||||||
fn wg_pubkey(privkey: &[u8]) -> Result<String> {
|
|
||||||
use std::io::Write as _;
|
|
||||||
use std::process::{Command, Stdio};
|
|
||||||
let mut child = Command::new("wg")
|
|
||||||
.arg("pubkey")
|
|
||||||
.stdin(Stdio::piped())
|
|
||||||
.stdout(Stdio::piped())
|
|
||||||
.stderr(Stdio::piped())
|
|
||||||
.spawn()
|
|
||||||
.context("run `wg pubkey` (is wireguard-tools installed?)")?;
|
|
||||||
child
|
|
||||||
.stdin
|
|
||||||
.take()
|
|
||||||
.context("wg pubkey: stdin unavailable")?
|
|
||||||
.write_all(privkey)
|
|
||||||
.context("write private key to `wg pubkey`")?;
|
|
||||||
let out = child.wait_with_output().context("wait for `wg pubkey`")?;
|
|
||||||
if !out.status.success() {
|
|
||||||
bail!(
|
|
||||||
"wg pubkey failed: {}",
|
|
||||||
String::from_utf8_lossy(&out.stderr).trim()
|
|
||||||
);
|
|
||||||
}
|
|
||||||
Ok(String::from_utf8_lossy(&out.stdout).trim().to_owned())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The closing comment of the `wg peer` snippet.
|
|
||||||
fn wg_peer_key_note() -> String {
|
|
||||||
format!(" # the key must be that hive's {HIVE_NAME_OPTION}")
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The first closing comment line of the `peer-config` snippet.
|
|
||||||
fn peer_config_key_note() -> String {
|
|
||||||
format!("# the key must be this hive's {HIVE_NAME_OPTION}, and the")
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `wg peer` — print the nix snippet to add a peer hive to the mesh.
|
|
||||||
/// Pure output (no fallible work), so it returns `()`; the dispatch arm
|
|
||||||
/// wraps it in `Ok` to match the sibling verbs.
|
|
||||||
pub(crate) fn wg_peer(domain: &str, pubkey: &str, address: &str, endpoint: Option<&str>) {
|
|
||||||
let key = hive_key(domain);
|
|
||||||
println!("Add to this hive's NixOS config:");
|
|
||||||
println!(" # `hives` describes the whole swarm and is meant to be the same");
|
|
||||||
println!(" # attrset on every host — add this entry to all of them.");
|
|
||||||
println!(" services.hyperhive.swarm.hives.\"{key}\" = {{");
|
|
||||||
println!(" domain = \"{domain}\";");
|
|
||||||
println!(" wireguardPublicKey = \"{pubkey}\";");
|
|
||||||
println!(" wireguardAddress = \"{address}\";");
|
|
||||||
if let Some(ep) = endpoint {
|
|
||||||
println!(" wireguardEndpoint = \"{ep}\";");
|
|
||||||
}
|
|
||||||
println!(" }};");
|
|
||||||
println!("{}", wg_peer_key_note());
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The `peer-config` step carrying the secret store's `pki` root: the
|
|
||||||
/// certificate itself, or where it will appear when this hive has not
|
|
||||||
/// fetched it yet.
|
|
||||||
fn store_root_step(pem: Option<&str>) -> String {
|
|
||||||
const HEAD: &str = "# 1. trust the SECRET STORE's pki root on the peer host (once, not per hive);\n\
|
|
||||||
# every hive's gateway leaf and the swarm's service names are issued under it:\n";
|
|
||||||
match pem.map(str::trim_end).filter(|p| !p.is_empty()) {
|
|
||||||
Some(p) => format!(
|
|
||||||
"{HEAD}{p}\n\
|
|
||||||
# (a self-signed hive in this swarm fetches it itself, into {STORE_PKI_ROOT_PATH})\n"
|
|
||||||
),
|
|
||||||
None => format!(
|
|
||||||
"{HEAD}# not on this host yet: swarm-services-cert.service writes it to\n\
|
|
||||||
# {STORE_PKI_ROOT_PATH}; rerun once that unit has succeeded\n"
|
|
||||||
),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `peer-config` — print the `swarm.hives."<name>"` block a peer
|
|
||||||
/// operator pastes to federate with THIS hive, plus the store-root step
|
|
||||||
/// when this hive serves a self-signed chain. Reads
|
|
||||||
/// local state only (the TLS trust bundle, the store root + the wg key);
|
|
||||||
/// prints, never mutates.
|
|
||||||
pub(crate) fn peer_config(domain: &str, wg_address: Option<&str>, wg_endpoint: Option<&str>) {
|
|
||||||
let self_signed = Path::new(HIVE_TLS_TRUST_BUNDLE_PATH).exists();
|
|
||||||
let key = hive_key(domain);
|
|
||||||
// WireGuard public key, when this hive has a mesh key. Best-effort:
|
|
||||||
// a missing key or absent `wg` binary just omits the mesh lines.
|
|
||||||
let wg_pub = std::fs::read(WG_KEY_PATH)
|
|
||||||
.ok()
|
|
||||||
.and_then(|k| wg_pubkey(&k).ok());
|
|
||||||
|
|
||||||
if self_signed {
|
|
||||||
let store_root = std::fs::read_to_string(STORE_PKI_ROOT_PATH).ok();
|
|
||||||
print!("{}", store_root_step(store_root.as_deref()));
|
|
||||||
println!();
|
|
||||||
println!("# 2. paste into the peer hive's NixOS config:");
|
|
||||||
} else {
|
|
||||||
println!("# paste into the peer hive's NixOS config:");
|
|
||||||
}
|
|
||||||
println!("services.hyperhive.swarm.hives.\"{key}\" = {{");
|
|
||||||
println!(" domain = \"{domain}\";");
|
|
||||||
if let Some(pk) = &wg_pub {
|
|
||||||
println!(" wireguardPublicKey = \"{pk}\";");
|
|
||||||
}
|
|
||||||
if let Some(addr) = wg_address {
|
|
||||||
println!(" wireguardAddress = \"{addr}\";");
|
|
||||||
}
|
|
||||||
if let Some(ep) = wg_endpoint {
|
|
||||||
println!(" wireguardEndpoint = \"{ep}\";");
|
|
||||||
}
|
|
||||||
println!("}};");
|
|
||||||
println!("{}", peer_config_key_note());
|
|
||||||
println!("# same entry belongs in every hive's config — `hives` is the swarm.");
|
|
||||||
if !self_signed {
|
|
||||||
println!(
|
|
||||||
"# (this hive's cert chains to a public CA — nothing to install; \
|
|
||||||
it's trusted by the default bundle.)"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `wg status` — show the live mesh interface (`wg show wg-hive`),
|
|
||||||
/// inheriting stdout so the operator sees it directly.
|
|
||||||
pub(crate) fn wg_status() -> Result<()> {
|
|
||||||
let status = std::process::Command::new("wg")
|
|
||||||
.args(["show", WG_INTERFACE])
|
|
||||||
.status()
|
|
||||||
.context("run `wg show` (is wireguard-tools installed?)")?;
|
|
||||||
if !status.success() {
|
|
||||||
bail!(
|
|
||||||
"`wg show {WG_INTERFACE}` failed — is the mesh enabled + up? \
|
|
||||||
(services.hyperhive.deploy.wireguard.enable = true, then deploy)"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::{
|
|
||||||
STORE_PKI_ROOT_PATH, peer_config_key_note, store_root_step, wg_peer_key_note,
|
|
||||||
write_key_0400,
|
|
||||||
};
|
|
||||||
|
|
||||||
const PEM: &str = "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----";
|
|
||||||
|
|
||||||
/// Both snippets tell the operator to set the deploy option, never the
|
|
||||||
/// removed `services.hyperhive` spelling.
|
|
||||||
#[test]
|
|
||||||
fn key_notes_name_the_deploy_option() {
|
|
||||||
for note in [wg_peer_key_note(), peer_config_key_note()] {
|
|
||||||
assert!(
|
|
||||||
note.contains("services.hyperhive.deploy.hive-controller.hiveName"),
|
|
||||||
"{note}"
|
|
||||||
);
|
|
||||||
assert!(!note.contains("services.hyperhive.hiveName"), "{note}");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The certificate is printed whole, on lines of its own, so a peer
|
|
||||||
/// operator can copy it out of the output as a PEM file.
|
|
||||||
#[test]
|
|
||||||
fn store_root_is_printed_verbatim() {
|
|
||||||
let s = store_root_step(Some(&format!("{PEM}\n")));
|
|
||||||
assert!(s.contains(&format!("\n{PEM}\n")), "{s}");
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A hive that has not fetched the root yet says where it will be,
|
|
||||||
/// rather than printing an empty step.
|
|
||||||
#[test]
|
|
||||||
fn missing_store_root_names_the_path_and_the_unit() {
|
|
||||||
for pem in [None, Some(""), Some("\n")] {
|
|
||||||
let s = store_root_step(pem);
|
|
||||||
assert!(s.contains(STORE_PKI_ROOT_PATH), "{s}");
|
|
||||||
assert!(s.contains("swarm-services-cert.service"), "{s}");
|
|
||||||
assert!(!s.contains("BEGIN CERTIFICATE"), "{s}");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The key reaches its path only by a rename from a sibling temp: with
|
|
||||||
/// the temp path blocked, the write fails and the existing key is left
|
|
||||||
/// byte-for-byte as it was. Once unblocked, the key lands at `0400` with
|
|
||||||
/// its bytes intact and no temp file beside it.
|
|
||||||
#[test]
|
|
||||||
fn key_is_written_through_a_temp_and_lands_0400() {
|
|
||||||
use std::os::unix::fs::PermissionsExt as _;
|
|
||||||
|
|
||||||
let dir = std::env::temp_dir().join(format!("hivectl-wg-key-{}", std::process::id()));
|
|
||||||
std::fs::create_dir_all(&dir).expect("mkdir");
|
|
||||||
let path = dir.join("hive.key");
|
|
||||||
let tmp = dir.join("hive.key.tmp");
|
|
||||||
std::fs::write(&path, "old-key\n").expect("seed");
|
|
||||||
std::fs::create_dir(&tmp).expect("block tmp");
|
|
||||||
|
|
||||||
let blocked = write_key_0400(&path, b"key-bytes\n");
|
|
||||||
assert!(blocked.is_err(), "a blocked temp must fail the write");
|
|
||||||
assert_eq!(std::fs::read(&path).expect("read seed"), b"old-key\n");
|
|
||||||
std::fs::remove_dir(&tmp).expect("unblock tmp");
|
|
||||||
|
|
||||||
write_key_0400(&path, b"key-bytes\n").expect("write");
|
|
||||||
let mode = std::fs::metadata(&path).expect("stat").permissions().mode() & 0o777;
|
|
||||||
let content = std::fs::read(&path).expect("read");
|
|
||||||
let entries: Vec<_> = std::fs::read_dir(&dir)
|
|
||||||
.expect("readdir")
|
|
||||||
.map(|e| e.expect("entry").file_name())
|
|
||||||
.collect();
|
|
||||||
std::fs::remove_dir_all(&dir).ok();
|
|
||||||
|
|
||||||
assert_eq!(mode, 0o400);
|
|
||||||
assert_eq!(content, b"key-bytes\n");
|
|
||||||
assert_eq!(entries, [std::ffi::OsString::from("hive.key")]);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -267,8 +267,7 @@ in
|
||||||
# nix/packages/default.nix + ./packages.nix) rather than the full
|
# nix/packages/default.nix + ./packages.nix) rather than the full
|
||||||
# `hyperhive` bundle — that bundle also carries `hivectl` (a
|
# `hyperhive` bundle — that bundle also carries `hivectl` (a
|
||||||
# host-admin CLI that dials the *host* admin socket — useless and
|
# host-admin CLI that dials the *host* admin socket — useless and
|
||||||
# unreachable from inside a container — wrapped with
|
# unreachable from inside a container). The daemon/harness/MCP bins
|
||||||
# `wireguard-tools` for `hivectl wg`). The daemon/harness/MCP bins
|
|
||||||
# the harness execs (hive-agent{,-mcp}, hive-bash-daemon,
|
# the harness execs (hive-agent{,-mcp}, hive-bash-daemon,
|
||||||
# hive-subagent-daemon, hive-matrix-daemon, hive-matrix-mcp) are
|
# hive-subagent-daemon, hive-matrix-daemon, hive-matrix-mcp) are
|
||||||
# wired via their own
|
# wired via their own
|
||||||
|
|
|
||||||
|
|
@ -97,7 +97,7 @@ in
|
||||||
environment.systemPackages = [
|
environment.systemPackages = [
|
||||||
# hive-forge <verb>: CLI wrapping common Forgejo REST API operations
|
# hive-forge <verb>: CLI wrapping common Forgejo REST API operations
|
||||||
# (view, pr, issue, comment, assign, close, labels, branches, etc.).
|
# (view, pr, issue, comment, assign, close, labels, branches, etc.).
|
||||||
# The per-bin split package — narrow closure, no hivectl/wireguard.
|
# The per-bin split package — narrow closure, no hivectl.
|
||||||
config.services.hyperhive.agent.packages.hive-forge
|
config.services.hyperhive.agent.packages.hive-forge
|
||||||
]
|
]
|
||||||
++ lib.optional (config.services.hyperhive.agent.forge.url != null) gitCredHelper;
|
++ lib.optional (config.services.hyperhive.agent.forge.url != null) gitCredHelper;
|
||||||
|
|
|
||||||
|
|
@ -50,7 +50,6 @@
|
||||||
./glue-nats-auth-bao-identity.nix
|
./glue-nats-auth-bao-identity.nix
|
||||||
./glue-nats-bao-identity.nix
|
./glue-nats-bao-identity.nix
|
||||||
./glue-nats-oidc-client.nix
|
./glue-nats-oidc-client.nix
|
||||||
./glue-nats-wireguard.nix
|
|
||||||
./glue-builder-oidc-client.nix
|
./glue-builder-oidc-client.nix
|
||||||
./glue-ci-metrics-otel-oidc-client.nix
|
./glue-ci-metrics-otel-oidc-client.nix
|
||||||
./glue-ci-runner-otel-oidc-client.nix
|
./glue-ci-runner-otel-oidc-client.nix
|
||||||
|
|
@ -95,7 +94,6 @@
|
||||||
./swarm-victorialogs.nix
|
./swarm-victorialogs.nix
|
||||||
./swarm-victoriametrics-service.nix
|
./swarm-victoriametrics-service.nix
|
||||||
./swarm-victoriametrics.nix
|
./swarm-victoriametrics.nix
|
||||||
./swarm-wireguard.nix
|
|
||||||
./swarm.nix
|
./swarm.nix
|
||||||
./hive-options-removed.nix
|
./hive-options-removed.nix
|
||||||
./host-ca-removed.nix
|
./host-ca-removed.nix
|
||||||
|
|
|
||||||
|
|
@ -74,6 +74,14 @@ let
|
||||||
The snapshot store was removed; agent-state backups are being rebuilt as
|
The snapshot store was removed; agent-state backups are being rebuilt as
|
||||||
swarm-archive. Remove this definition.
|
swarm-archive. Remove this definition.
|
||||||
'';
|
'';
|
||||||
|
|
||||||
|
# Shared by the removed options of the WireGuard mesh.
|
||||||
|
wireguardRemoved = ''
|
||||||
|
The WireGuard mesh was removed. bao and nats are reached through
|
||||||
|
their routable listeners (`services.hyperhive.deploy.bao.extraListenAddresses`
|
||||||
|
with `openFirewall`, and `services.hyperhive.deploy.nats.openFirewall`; both
|
||||||
|
off by default). Remove this definition.
|
||||||
|
'';
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
imports = [
|
imports = [
|
||||||
|
|
@ -181,6 +189,41 @@ in
|
||||||
"snapshotStore"
|
"snapshotStore"
|
||||||
"port"
|
"port"
|
||||||
] snapshotStoreRemoved)
|
] snapshotStoreRemoved)
|
||||||
|
(lib.mkRemovedOptionModule [
|
||||||
|
"services"
|
||||||
|
"hyperhive"
|
||||||
|
"deploy"
|
||||||
|
"wireguard"
|
||||||
|
"enable"
|
||||||
|
] wireguardRemoved)
|
||||||
|
(lib.mkRemovedOptionModule [
|
||||||
|
"services"
|
||||||
|
"hyperhive"
|
||||||
|
"deploy"
|
||||||
|
"wireguard"
|
||||||
|
"privateKeyFile"
|
||||||
|
] wireguardRemoved)
|
||||||
|
(lib.mkRemovedOptionModule [
|
||||||
|
"services"
|
||||||
|
"hyperhive"
|
||||||
|
"deploy"
|
||||||
|
"wireguard"
|
||||||
|
"address"
|
||||||
|
] wireguardRemoved)
|
||||||
|
(lib.mkRemovedOptionModule [
|
||||||
|
"services"
|
||||||
|
"hyperhive"
|
||||||
|
"deploy"
|
||||||
|
"wireguard"
|
||||||
|
"listenPort"
|
||||||
|
] wireguardRemoved)
|
||||||
|
(lib.mkRemovedOptionModule [
|
||||||
|
"services"
|
||||||
|
"hyperhive"
|
||||||
|
"deploy"
|
||||||
|
"wireguard"
|
||||||
|
"persistentKeepalive"
|
||||||
|
] wireguardRemoved)
|
||||||
|
|
||||||
# Same type, same meaning, new path — so a rename carries it exactly
|
# Same type, same meaning, new path — so a rename carries it exactly
|
||||||
# and existing configs keep evaluating with one warning naming both
|
# and existing configs keep evaluating with one warning naming both
|
||||||
|
|
|
||||||
|
|
@ -1,21 +0,0 @@
|
||||||
# Glue: where the queue and the wireguard mesh share a host, the queue's port
|
|
||||||
# is open on the mesh interface.
|
|
||||||
#
|
|
||||||
# Interface-scoped, never host-wide: the firewall is default-deny before a
|
|
||||||
# packet reaches the listener, so without this no other hive can reach the
|
|
||||||
# queue. Gated on BOTH being here — the queue's own module gate
|
|
||||||
# (`deploy.nats.enable`) is restated below; a change there must change here.
|
|
||||||
{
|
|
||||||
lib,
|
|
||||||
config,
|
|
||||||
...
|
|
||||||
}:
|
|
||||||
let
|
|
||||||
hyperhiveCfg = config.services.hyperhive;
|
|
||||||
deployCfg = hyperhiveCfg.deploy;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
config = lib.mkIf (deployCfg.nats.enable && deployCfg.wireguard.enable) {
|
|
||||||
networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ hyperhiveCfg.swarm.nats.port ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
@ -40,9 +40,6 @@ let
|
||||||
nft = lib.getExe pkgs.nftables;
|
nft = lib.getExe pkgs.nftables;
|
||||||
br = cfg.bridgeName;
|
br = cfg.bridgeName;
|
||||||
|
|
||||||
# The mesh interface (swarm-wireguard.nix): inter-hive traffic is not egress.
|
|
||||||
meshInterface = "wg-hive";
|
|
||||||
|
|
||||||
# With `refuseEch`, the redirect of port 443 ends at `hive-priv
|
# With `refuseEch`, the redirect of port 443 ends at `hive-priv
|
||||||
# egress-ech-relay`, which refuses a ClientHello that carries the ECH
|
# egress-ech-relay`, which refuses a ClientHello that carries the ECH
|
||||||
# extension and sends the rest on to squid's HTTPS intercept port. The relay
|
# extension and sends the rest on to squid's HTTPS intercept port. The relay
|
||||||
|
|
@ -143,8 +140,8 @@ let
|
||||||
|
|
||||||
# IPv4 TCP from the agent bridge to ports 80 and 443. `fib daddr oifname` is
|
# IPv4 TCP from the agent bridge to ports 80 and 443. `fib daddr oifname` is
|
||||||
# the interface the route to the destination leaves by: the bridge (another
|
# the interface the route to the destination leaves by: the bridge (another
|
||||||
# container), `lo` (an address of the host: `fib daddr type local`) and the
|
# container) and `lo` (an address of the host: `fib daddr type local`) are
|
||||||
# mesh interface are not egress. The redirect rewrites the destination to
|
# not egress. The redirect rewrites the destination to
|
||||||
# the bridge address and the port of squid's intercept listener (for port 443
|
# the bridge address and the port of squid's intercept listener (for port 443
|
||||||
# with `refuseEch`, of the relay), which reads the original destination from
|
# with `refuseEch`, of the relay), which reads the original destination from
|
||||||
# conntrack. IPv6 is left alone: the agent bridge has no IPv6 addressing.
|
# conntrack. IPv6 is left alone: the agent bridge has no IPv6 addressing.
|
||||||
|
|
@ -153,7 +150,7 @@ let
|
||||||
type nat hook prerouting priority dstnat; policy accept;
|
type nat hook prerouting priority dstnat; policy accept;
|
||||||
iifname "${br}" meta nfproto ipv4 ether saddr @direct_mac return
|
iifname "${br}" meta nfproto ipv4 ether saddr @direct_mac return
|
||||||
iifname "${br}" meta nfproto ipv4 fib daddr type local return
|
iifname "${br}" meta nfproto ipv4 fib daddr type local return
|
||||||
iifname "${br}" meta nfproto ipv4 fib daddr oifname { "${br}", "${meshInterface}" } return
|
iifname "${br}" meta nfproto ipv4 fib daddr oifname "${br}" return
|
||||||
iifname "${br}" meta nfproto ipv4 tcp dport 80 counter redirect to :${toString egressCfg.intercept.httpPort}
|
iifname "${br}" meta nfproto ipv4 tcp dport 80 counter redirect to :${toString egressCfg.intercept.httpPort}
|
||||||
iifname "${br}" meta nfproto ipv4 tcp dport 443 counter redirect to :${
|
iifname "${br}" meta nfproto ipv4 tcp dport 443 counter redirect to :${
|
||||||
toString (if refuseEch then intercept.relayPort else intercept.httpsPort)
|
toString (if refuseEch then intercept.relayPort else intercept.httpsPort)
|
||||||
|
|
@ -171,8 +168,8 @@ let
|
||||||
# Priority `filter - 10` runs before the NixOS firewall's forward handling,
|
# Priority `filter - 10` runs before the NixOS firewall's forward handling,
|
||||||
# which sits at `filter` on both backends (`FORWARD` of the iptables backend,
|
# which sits at `filter` on both backends (`FORWARD` of the iptables backend,
|
||||||
# the `forward` chain of `nixos-fw`). An `accept` here ends this chain only,
|
# the `forward` chain of `nixos-fw`). An `accept` here ends this chain only,
|
||||||
# so it allows nothing those refuse; a `reject` ends the packet. The mesh
|
# so it allows nothing those refuse; a `reject` ends the packet. The MACs
|
||||||
# and the MACs in `direct_mac` (the unrestricted agents and the service
|
# in `direct_mac` (the unrestricted agents and the service
|
||||||
# containers, the traffic the intercept chain leaves alone) come first in
|
# containers, the traffic the intercept chain leaves alone) come first in
|
||||||
# both modes, so a `log` line names what `reject` would refuse. Traffic to
|
# both modes, so a `log` line names what `reject` would refuse. Traffic to
|
||||||
# the host's own addresses (squid, dnsmasq, the gateway) is delivered
|
# the host's own addresses (squid, dnsmasq, the gateway) is delivered
|
||||||
|
|
@ -183,7 +180,6 @@ let
|
||||||
|
|
||||||
chain forward {
|
chain forward {
|
||||||
type filter hook forward priority filter - 10; policy accept;
|
type filter hook forward priority filter - 10; policy accept;
|
||||||
iifname "${br}" oifname "${meshInterface}" accept
|
|
||||||
${fromBridge} ether saddr @direct_mac accept
|
${fromBridge} ether saddr @direct_mac accept
|
||||||
${fromBridge} counter
|
${fromBridge} counter
|
||||||
${lib.concatStringsSep "\n " (
|
${lib.concatStringsSep "\n " (
|
||||||
|
|
@ -315,7 +311,7 @@ in
|
||||||
default = "log";
|
default = "log";
|
||||||
description = ''
|
description = ''
|
||||||
What the `inet hive_egress` table does with traffic the host forwards
|
What the `inet hive_egress` table does with traffic the host forwards
|
||||||
from the agent bridge to anywhere but the bridge and the mesh. In `log` and
|
from the agent bridge to anywhere but the bridge. In `log` and
|
||||||
`reject` the traffic of the MACs in the `direct_mac` set (the unrestricted
|
`reject` the traffic of the MACs in the `direct_mac` set (the unrestricted
|
||||||
agents and the service containers) is accepted and never logged.
|
agents and the service containers) is accepted and never logged.
|
||||||
|
|
||||||
|
|
|
||||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue