hive-matrix: run the homeserver in its own netns, behind a unix socket
tuwunel bound `0.0.0.0:<httpPort>` in the host netns, so the homeserver
port was open on every host address, and the gateway proxied to it on
host loopback.
The container now sets `privateNetwork = true` and hangs off the hive
bridge with a DHCP lease, the same attach the CI runner, builder and
review classifier use (resolvconf on, `networking.nameservers` the bridge
IP). tuwunel listens only on `/run/hive-matrix/tuwunel.sock`, a host
directory (root:nginx 0770) bind-mounted into the container; its primary
group is nginx's gid and the socket is 0660. The gateway's `/_matrix/`
location proxies to the socket.
The static resolv.conf goes: it existed because resolvconf's host
tracking came up empty in the shared netns.
`swarm.matrix.httpPort` and `{swarm,deploy}.matrix.openFirewall` are
removed with a message: there is no host port to choose or open. The
`.well-known/matrix/*` fallback to `<hive>:<httpPort>` is dropped with
them; it only rendered with `gatewayHost = null`, which the homeserver's
SSO assertion already rejects.
Part of #4815.
This commit is contained in:
atlas
mara
parent
e455964cb3
commit
34e142bb5c
11 changed files with 191 additions and 163 deletions
|
|
@ -6,9 +6,9 @@ client at `chat.<swarm-domain>/` (the `gatewayHost` vhost). A swarm runs one
|
|||
homeserver, on one host. Two namespaces configure it:
|
||||
|
||||
- `services.hyperhive.swarm.matrix.*` — what the homeserver **is**, as
|
||||
every hive sees it: `serverName`, `gatewayHost`, ports, `allowEncryption`.
|
||||
every hive sees it: `serverName`, `gatewayHost`, `apiUrl`, `allowEncryption`.
|
||||
- `services.hyperhive.deploy.matrix.*` — what the host running it decides:
|
||||
`enable`, `gui.package`, `openFirewall`, `trustedServers`,
|
||||
`enable`, `gui.package`, `trustedServers`,
|
||||
`maxRequestSize`, `sso.clientSecretFile`.
|
||||
|
||||
Vhost routing lives in [`gateway.md`](../networking/gateway.md).
|
||||
|
|
@ -23,10 +23,10 @@ Same shape as [`gateway.md::hive-forge container shape`](../networking/gateway.m
|
|||
- Keeps hive-matrix from fighting any `services.matrix-*` the
|
||||
operator already runs on the host — separate systemd namespace,
|
||||
separate state dir.
|
||||
- Container shares the host network namespace
|
||||
(`privateNetwork = false`) for state + systemd-unit isolation. Agents
|
||||
reach the homeserver at `chat.<swarm-domain>` via the gateway (agents
|
||||
run in private netns and can't access host loopback directly).
|
||||
- Container runs in a network namespace of its own
|
||||
(`privateNetwork = true`) on the hive bridge, and binds no TCP port. Agents
|
||||
reach the homeserver at `chat.<swarm-domain>` via the gateway, which
|
||||
dials its unix socket (see [Reach](#reach)).
|
||||
- Persistent state at
|
||||
`/var/lib/nixos-containers/hive-matrix/var/lib/matrix-tuwunel/`
|
||||
survives container restart / host reboot. To wipe, destroy the
|
||||
|
|
@ -44,7 +44,7 @@ Two distinct hostnames:
|
|||
- **`gatewayHost`** — the API listener hostname, where the gateway's
|
||||
matrix vhost proxies `/_matrix/*` to tuwunel. Defaults to
|
||||
`chat.<services.hyperhive.swarm.domain>`. Set to `null` to skip the
|
||||
gateway vhost (tuwunel stays direct on `httpPort`).
|
||||
gateway vhost, which leaves nothing able to reach the homeserver.
|
||||
|
||||
Both default under the **swarm** domain, because a swarm runs one
|
||||
homeserver: tying its identity to a single hive's domain would make
|
||||
|
|
@ -80,14 +80,15 @@ pin the value before the homeserver mints any ids under the default.
|
|||
|
||||
</details>
|
||||
|
||||
## Default-closed firewall
|
||||
## Reach
|
||||
|
||||
`deploy.matrix.openFirewall` defaults to `false`: hive-c0re and agent
|
||||
containers reach the homeserver at `chat.<swarm-domain>` via the gateway,
|
||||
so the firewall hole only matters for direct access to `httpPort` from
|
||||
*outside* the host. Flip to `true` when announcing the homeserver to
|
||||
other hives or when an external matrix client needs to reach the
|
||||
client-server API directly.
|
||||
tuwunel binds no TCP port. It listens on the unix socket
|
||||
`/run/hive-matrix/tuwunel.sock`, in a host directory (`root:nginx`,
|
||||
`0770`) bind-mounted into the container, and the gateway's
|
||||
`/_matrix/` location proxies to that socket. Clients, agents, hive-c0re
|
||||
and federation peers all reach the homeserver at `chat.<swarm-domain>`
|
||||
through the gateway; from outside the host that takes
|
||||
`gateway.openFirewall`.
|
||||
|
||||
hive-c0re provisions matrix (its own sender login, the hive Space and
|
||||
chat room, invites) through `swarm.matrix.apiUrl`, which defaults to
|
||||
|
|
@ -96,11 +97,15 @@ the hive's matrix access token, and on a hive that doesn't run the
|
|||
homeserver it crosses the network, so the default is TLS; hive-c0re
|
||||
verifies the gateway's certificate against the hive trust bundle.
|
||||
|
||||
Federation port 8448 is intentionally not opened here — tuwunel
|
||||
serves the federation API on the same `httpPort` as client-server
|
||||
by default. Reaching it on 8448 needs either an explicit tuwunel
|
||||
bind to that port OR a reverse-proxy + `.well-known/matrix/server`
|
||||
delegation (the latter lives in `gateway.md::Discovery flow`).
|
||||
Federation peers find the homeserver through `.well-known/matrix/server`
|
||||
on the hive domain, which names `<gatewayHost>:<httpsPort>`
|
||||
([`gateway.md::Discovery flow`](../networking/gateway.md#discovery-flow-matrix)).
|
||||
Port 8448 isn't used.
|
||||
|
||||
Outbound, the container sits on the hive bridge like the CI runner: it
|
||||
leases an address from the gateway's DHCP, resolves through its dnsmasq,
|
||||
and leaves through the bridge's NAT. It reaches the secret store,
|
||||
authelia and federation peers by name.
|
||||
|
||||
## Provisioning flow (appservice)
|
||||
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ You rarely switch it on yourself. `gateway.enable` defaults to off, and every mo
|
|||
| `auth.<swarm>/` | authelia (`9091`) | `swarm-authelia.nix`, `deploy.authelia.enable` |
|
||||
| `auth.<swarm>/bridge/*` | swarm-authelia-bridge (`9092`), prefix stripped; answers swarm-controller's client only | `swarm-authelia.nix`, `deploy.authelia.enable` |
|
||||
| `forge.<swarm>/` | forgejo (`3000`) | `hive-forge/`, `deploy.forgejo.enable` |
|
||||
| `chat.<swarm>/_matrix/*` | tuwunel (`8008`) | `hive-matrix.nix`, `swarm.matrix.gatewayHost != null` |
|
||||
| `chat.<swarm>/_matrix/*` | tuwunel (unix socket `/run/hive-matrix/tuwunel.sock`) | `hive-matrix.nix`, `swarm.matrix.gatewayHost != null` |
|
||||
| `chat.<swarm>/` | fluffychat-web static | `hive-matrix.nix`, `swarm.matrix.gatewayHost != null` |
|
||||
| `chat.<swarm>/config.json` | inline JSON (FluffyChat boot config) | `hive-matrix.nix`, `swarm.matrix.gatewayHost != null` |
|
||||
| `<queue domain>/metrics` | the queue exporter's `/metrics` on loopback, behind an authelia `auth_request`; every other path, `/` included, returns `403`, so `/varz` and `/connz` are never reachable | `swarm-nats.nix`, the queue's host |
|
||||
|
|
@ -221,7 +221,7 @@ The operator points a client at `<hive>`:
|
|||
|
||||
1. The client fetches `https://<hive>/.well-known/matrix/client` → `{"m.homeserver":{"base_url":"https://chat.<swarm>"}}` (no port suffix when `httpsPort` is 443).
|
||||
2. It connects to `chat.<swarm>/_matrix/client/...`.
|
||||
3. The gateway proxies `/_matrix/*` to tuwunel at `127.0.0.1:8008`.
|
||||
3. The gateway proxies `/_matrix/*` to tuwunel over its unix socket, `/run/hive-matrix/tuwunel.sock`.
|
||||
|
||||
matrix-dart-sdk (FluffyChat and others) always fetches the well-known over `https`, which is why [self-signed TLS](#self-signed-tls-default) is on by default.
|
||||
|
||||
|
|
|
|||
|
|
@ -22,12 +22,11 @@ that touches it.
|
|||
│ uplink NIC — NAT MASQUERADE for the
|
||||
│ bridge subnet (10.42.0.0/24 default)
|
||||
┌──────────────────────────┴─────────────────────────────────────────┐
|
||||
│ host netns — the host itself plus gateway / forge / matrix │
|
||||
│ host netns — the host itself plus gateway / forge │
|
||||
│ │
|
||||
│ nginx :80/:443 [hive-gateway] │
|
||||
│ dnsmasq 10.42.0.1:53 (DNS) + :67 (DHCP) [hive-gateway] │
|
||||
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
|
||||
│ tuwunel :8008 client API [hive-matrix] │
|
||||
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
|
||||
│ wg-hive :51820/udp — swarm mesh, when enabled (host iface) │
|
||||
│ │
|
||||
|
|
@ -46,7 +45,7 @@ that touches it.
|
|||
| ------------------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| gateway (host) | host | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
|
||||
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
|
||||
| `hive-matrix` | host (shared) | host addresses | tuwunel `:8008` (+ optional federation port); fronted by the matrix vhost |
|
||||
| `hive-matrix` | private, veth on bridge | DHCP pool | tuwunel on UDS `/run/hive-matrix/tuwunel.sock` → the matrix vhost; out to store, SSO, peers |
|
||||
| `swarm-ci-runner` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
|
||||
| `swarm-review-classifier` | private, veth on bridge | DHCP pool | outbound only (classifier → queue, secret store, hive collector); no inbound surface |
|
||||
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
|
||||
|
|
@ -64,7 +63,7 @@ The flows, end to end:
|
|||
- **HTTP** — `forge.` and `chat.` (under `swarm.domain`) plus the hive's
|
||||
own dashboard name resolve to the bridge IP, land on nginx
|
||||
`:80`/`:443`, and proxy to forgejo
|
||||
`:3000`, tuwunel `:8008`, hive-c0re `127.0.0.1:7000`, or a per-agent
|
||||
`:3000`, tuwunel's unix socket, hive-c0re `127.0.0.1:7000`, or a per-agent
|
||||
UI unix socket.
|
||||
- **Internet egress** — agent default route points at the bridge IP;
|
||||
the host forwards + masquerades out its uplink.
|
||||
|
|
|
|||
Loading…
Reference in a new issue