From cd786ef4014d8e78a7ca432f5bb1b1aff5f1ebf1 Mon Sep 17 00:00:00 2001 From: atlas Date: Sun, 15 Nov 2026 16:15:29 +0100 Subject: [PATCH] document the controller tcp listener and its two leaves --- docs/4-implementation/networking/gateway.md | 7 +++++ docs/4-implementation/swarm/ca.md | 30 +++++++++++++++++++ docs/4-implementation/swarm/controller.md | 27 +++++++++++++++++ .../swarm/credentials-reference.md | 2 ++ nix/host-modules/swarm-controller.nix | 3 +- swarm-controller/README.md | 15 ++++++++++ 6 files changed, 83 insertions(+), 1 deletion(-) diff --git a/docs/4-implementation/networking/gateway.md b/docs/4-implementation/networking/gateway.md index 20331da4d..f082e2cc2 100644 --- a/docs/4-implementation/networking/gateway.md +++ b/docs/4-implementation/networking/gateway.md @@ -472,6 +472,13 @@ the opposite arm run as a control: that's every service on the hive. - `server_name on` — sends SNI, or the far end can't pick a cert. +`verifiedProxyToWithClientCert name { certificate, key }` is the same block plus +`proxy_ssl_certificate` and `proxy_ssl_certificate_key`, for an upstream that demands a client +certificate. The swarm UI's `/api/` and `/webhook/forge/` locations use it when the swarm +controller's TCP listener is on (`swarm.controller.tcp.enable`); `certificate` and `key` are host +paths nginx can read. `verifiedProxyTo` itself presents no client certificate, and every other +caller keeps rendering exactly what it did. + **⚠️ Session-cache footgun**: the module leaves `proxy_ssl_session_reuse` at its default (on), deliberately — nginx uses this on per-request auth subrequests, so every request pays the handshake it avoids. Worth diff --git a/docs/4-implementation/swarm/ca.md b/docs/4-implementation/swarm/ca.md index e3ef5165e..88ba09b12 100644 --- a/docs/4-implementation/swarm/ca.md +++ b/docs/4-implementation/swarm/ca.md @@ -206,6 +206,36 @@ and the next day's run tries again: about fifteen tries before the leaf lapses. `systemctl list-timers swarm-services-cert-renew` shows the next run. +## The controller's TCP listener has two leaves of its own + +`swarm.controller.tcp.enable` adds a TLS listener to the swarm controller (default off). The +controller checks nobody per request, so the listener finishes a handshake for one client only: +the gateway. Both ends hold a leaf from the store's `pki` mount, each through a role of its own on +the same mount and the same root: + +- **Server leaf.** The `swarm-controller-tls` role issues for `swarm.controller.tcp.domain` alone, + `serverAuth` only. `swarm-controller-tls.service` on the controller's host requests it under the + controller's own store leaf, whose policy gains that one issue path. +- **Gateway client leaf.** The `swarm-controller-gateway-client` role issues for + `gateway-client.` alone, `clientAuth` only. `swarm-controller-gateway-client.service` + on the host that runs the controller requests it under the services-issuer identity, whose policy + gains that one issue path. + +The controller accepts a client certificate that chains to the services root, carries the +`clientAuth` usage and names `gateway-client.`. The swarm-services leaf chains to +the same root but is `serverAuth` only and names other hosts, so the listener refuses it at the +handshake. Nginx verifies the controller's server leaf against the trust bundle, by `domain`. + +Every host that holds the services-issuer identity can obtain the gateway client leaf, which is +every gateway host. The controller trusts exactly that identity: a gateway of this swarm. + +Set `swarm.controller.tcp.enable` and `swarm.controller.tcp.domain` identically on every host. The +store's host writes both roles from them. A store host that leaves the option off has no role, and +the issuing unit fails with the store's refusal naming it. + +The listener reads the services root once at start. Replacing the root (about 30 years out) needs a +controller restart as well as new leaves. + ## Constraints on the material No CA private key exists on any hive. The `pki` root's key stays inside the diff --git a/docs/4-implementation/swarm/controller.md b/docs/4-implementation/swarm/controller.md index bee6a590a..d5ab061c9 100644 --- a/docs/4-implementation/swarm/controller.md +++ b/docs/4-implementation/swarm/controller.md @@ -498,6 +498,33 @@ The controller registers the hooks within five minutes of having both the forge token and the secret. It keeps trying, at most five minutes apart, for as long as the forge or the secret store is down. A restart isn't needed to recover. +The URL hooks register against is `deploy.swarm-controller.publicUrl`, default +`https://`. It names the vhost that fronts the controller, so it stays right +whether the gateway reaches the controller by socket or by its TCP listener. Set it when another +URL fronts the controller. + +## Reaching the controller over TCP (optional) + +By default the gateway reaches the controller through a unix socket on the same host. Setting +`swarm.controller.tcp.enable` and `swarm.controller.tcp.domain` on **every host of the swarm** +adds a TLS listener beside the socket and points the gateway's `/api/` and `/webhook/forge/` +locations at it by that name. Nothing else changes: the socket stays, and the operator gate on +`/api/` stays. + +What you provide: + +- `swarm.controller.tcp.domain`, resolvable from the gateway's host. Nginx resolves it when it + loads its config. +- `deploy.swarm-controller.tcp.openFirewall = true` on the controller's host only when a gateway + on another host dials it. The port (`swarm.controller.tcp.port`, default 9443) stays closed + until you do, and `deploy.swarm-controller.tcp.listenAddress` picks the bound address. + +What you don't: both certificates come from the secret store with no manual step +([`ca.md`](ca.md#the-controllers-tcp-listener-has-two-leaves-of-its-own)). The listener accepts +the gateway's client certificate and nothing else, so reaching the port gets a stranger a refused +handshake. The swarm UI's vhost is still declared on the host that runs the controller, so the +gateway that dials the listener runs there too. + ## Agent identity minting A store identity that fails to mint logs `swarm jobq: mint_agent_identity diff --git a/docs/4-implementation/swarm/credentials-reference.md b/docs/4-implementation/swarm/credentials-reference.md index 0fb35409f..61bdb96ad 100644 --- a/docs/4-implementation/swarm/credentials-reference.md +++ b/docs/4-implementation/swarm/credentials-reference.md @@ -152,6 +152,8 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](../../2-hand | queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place | +| the controller's TCP listener certificate and key | `swarm-controller-tls.service` on the controller's host, only where you set `swarm.controller.tcp.enable`. It logs in under the controller's own store leaf and issues from the store's `swarm-controller-tls` role: one name, `swarm.controller.tcp.domain`, `serverAuth` only. A daily timer renews it past half its life | `/{cert,key}.pem` (`/var/lib/swarm-controller-tls`), the key `0640` group `swarm-controller`; the daemon re-reads both when they change, so a rotation restarts nothing. A store host that leaves the option off has no role to issue from, and the unit fails with the store's refusal naming it | Nothing to place: the controller's own store leaf is the one it already holds | +| the gateway's client certificate for the controller's TCP listener | `swarm-controller-gateway-client.service` on the host that runs the controller, only where you set `swarm.controller.tcp.enable`. It logs in under the services-issuer identity and issues from the store's `swarm-controller-gateway-client` role: one name, `gateway-client.`, `clientAuth` only, so the services leaf can't stand in for it. A daily timer renews it | `/controller-client.pem` and `controller-client-key.pem`, the key `0640` group `nginx`; the gateway's controller locations name them in `proxy_ssl_certificate`. Until the store answers, a throwaway self-signed pair keeps nginx's config loadable. The listener refuses that pair, so those locations answer 502. Every holder of the services-issuer identity can obtain this leaf: the listener trusts "a gateway of this swarm" | Nothing to place: the services-issuer identity is the one every gateway host already holds | | the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | | the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | diff --git a/nix/host-modules/swarm-controller.nix b/nix/host-modules/swarm-controller.nix index 11133bd98..afeef53aa 100644 --- a/nix/host-modules/swarm-controller.nix +++ b/nix/host-modules/swarm-controller.nix @@ -626,7 +626,8 @@ in readOnly = true; description = '' Unix socket the daemon serves on, and the path the gateway's nginx - proxies to. + proxies to unless + `swarm.controller.tcp.enable` points it at the TCP listener. The **directory** is the access control here, not the socket mode: the socket itself is `0666`, exactly as hive-c0re publishes the diff --git a/swarm-controller/README.md b/swarm-controller/README.md index 72ff77dbe..0c9e348cc 100644 --- a/swarm-controller/README.md +++ b/swarm-controller/README.md @@ -141,6 +141,21 @@ The socket path is `services.hyperhive.deploy.swarm-controller.socketPath`, defa `/run/swarm-controller/controller.sock`, exported to the process as `SWARM_CONTROLLER_SOCKET`. +## The optional TCP listener + +`services.hyperhive.swarm.controller.tcp.enable` (default off) adds a TLS listener beside the +socket, for a gateway that reaches the controller over the network. The routes trust their +transport and check no caller, so the listener finishes a handshake for one client only +(`src/tcp_tls.rs`): a certificate that chains to the swarm services root, carries `clientAuth`, +and names `gateway-client.` in a subject alternative name. No certificate, another +name, or the `serverAuth`-only swarm-services leaf is refused during the handshake, before any +HTTP. The socket stays and answers exactly as before. + +The daemon takes five variables, all or none (`SWARM_CONTROLLER_TCP_LISTEN`, `_CERT`, `_KEY`, +`_CLIENT_CA`, `_CLIENT_NAME`), which the nix module sets from the options. It re-reads the server +certificate and key when either file changes, so a rotation needs no restart; it reads the client CA +once at start. [`ca.md`](../docs/4-implementation/swarm/ca.md) covers where both leaves come from. + ## ⚠️ The socket's directory is its access control The socket is `0666`. It has to be: nginx runs as a different user and