remove the swarm wireguard mesh; tombstone its options
This commit is contained in:
atlas
parent
fbfc6f392a
commit
adc61c9043
4 changed files with 330 additions and 1068 deletions
|
|
@ -29,7 +29,6 @@ that touches it.
|
|||
│ squid 10.42.0.1:3128 (egress proxy) [hive-squid] │
|
||||
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
|
||||
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
|
||||
│ wg-hive :51820/udp — swarm mesh, when enabled (host iface) │
|
||||
│ │
|
||||
│ hive-br0 10.42.0.1/24 │
|
||||
│ ┌──────────┼──────────────┐ │
|
||||
|
|
@ -72,8 +71,8 @@ The flows, end to end:
|
|||
UI unix socket.
|
||||
- **Internet egress** — agent default route points at the bridge IP;
|
||||
the host forwards + masquerades out its uplink.
|
||||
- **Swarm** — peer hives connect over the `wg-hive` WireGuard mesh
|
||||
and reach each other's gateway/forge across it
|
||||
- **Swarm** — peer hives reach each other's gateway and forge over HTTPS, and
|
||||
the queue and the secret store on their routable listeners
|
||||
([`docs/2-handbook/swarm/`](../../2-handbook/swarm/README.md)).
|
||||
- **Control plane (no network)** — per-agent broker socket
|
||||
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
|
||||
|
|
@ -526,7 +525,7 @@ squid (`services.hyperhive.deploy.hive-controller.egress.intercept.enable`).
|
|||
|
||||
- **Who gets redirected.** Every flow from the bridge to port 80 or 443, except
|
||||
a flow to an address that stays on the bridge, to one of the host's own
|
||||
addresses or to the `wg-hive` interface, and a flow from a MAC in the set
|
||||
addresses, and a flow from a MAC in the set
|
||||
`direct_mac`. That set holds the MACs of the ports of every container that's
|
||||
not an agent (the swarm's service containers) and of the agents listed in
|
||||
`services.hyperhive.deploy.hive-controller.squid.unrestrictedFile`. So
|
||||
|
|
@ -727,13 +726,13 @@ loopback — the hive-c0re admin socket is a UDS, not TCP.
|
|||
|
||||
`services.hyperhive.deploy.hive-controller.egress.forward` sets what the `forward` chain of
|
||||
the `inet hive_egress` table does with traffic the host routes from the agent
|
||||
bridge to anywhere but the bridge and the `wg-hive` mesh interface. It takes
|
||||
bridge to anywhere but the bridge. It takes
|
||||
`accept`, `log` (the default) or `reject`. In `log` mode the chain blocks
|
||||
nothing. It logs each new flow and counts every packet. In `reject` mode it
|
||||
also refuses the flow.
|
||||
|
||||
- **Who is exempt.** In `log` and `reject` mode alike, the chain accepts the
|
||||
mesh traffic first, then the traffic of every MAC in the `direct_mac` set: the
|
||||
traffic of every MAC in the `direct_mac` set: the
|
||||
`unrestricted` agents and the service containers, which the `intercept` chain
|
||||
leaves alone too (see _Intercepted web traffic_). They leave no line, so a
|
||||
`hive-egress-would-drop` line is exactly a flow that `reject` would refuse,
|
||||
|
|
|
|||
|
|
@ -128,9 +128,7 @@ by hand, start the workflow from the forge UI (Actions, vm-nightly, Run
|
|||
workflow, optional `test` input), or run
|
||||
`nix build .#vmTests.x86_64-linux.<name>` on a host with KVM. Current tests:
|
||||
`agent-ephemeral-rootfs` (the switch of an agent container to an ephemeral
|
||||
root, through the real hive-priv unit), `hive-squid`, and `wg-hive` (two hives
|
||||
peered over the `wg-hive` mesh: handshake, traffic, and a wrong-key peer that
|
||||
must not handshake).
|
||||
root, through the real hive-priv unit) and `hive-squid`.
|
||||
|
||||
The `full-setup` script is plain Python in `nix/vm-tests/full-setup.py`, with no
|
||||
Nix interpolation. Commands the script runs in the VM find their binaries on the
|
||||
|
|
|
|||
|
|
@ -72,9 +72,9 @@ The queue URL is the same string on every hive. The queue accepts TLS only,
|
|||
with a certificate for `swarm.nats.domain` (default `nats.<swarm.domain>`) and
|
||||
no other name or address, so a URL with an IP address or `nats://` fails.
|
||||
The queue's host resolves the name itself. **A multi-host swarm needs one
|
||||
upstream DNS record**, `nats.<swarm.domain>` pointing at the queue host's mesh
|
||||
upstream DNS record**, `nats.<swarm.domain>` pointing at the queue host's routable
|
||||
address, the same contract as `bao.<swarm.domain>`. The queue's port is open
|
||||
on `wg-hive` when that host is on the mesh.
|
||||
host-wide only with `deploy.nats.openFirewall`.
|
||||
|
||||
The queue's host asks the secret store for that certificate under a store
|
||||
leaf of its own, `deploy.nats.baoClientCertFile` / `baoClientKeyFile`. On the
|
||||
|
|
|
|||
|
|
@ -148,7 +148,6 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](../../2-hand
|
|||
| the CI run poller's copy of its OIDC secret | authelia's mint, which creates it in the swarm secret store | `swarm-ci-metrics` reads `swarm/services/<swarm.ci-metrics.clientId>/oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else |
|
||||
| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's secret (`swarm.authelia.otel.clientId`). It creates it at `swarm/services/<id>/oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place |
|
||||
|
|
|
|||
Loading…
Reference in a new issue