swarm-bao: reach the store through a TLS passthrough, not a vhost

An agent container cannot dial the store's loopback listener: the bridge
to-loopback DROP rule is there precisely to stop that, and the store
authenticates every reader by client certificate, so the usual answer —
a gateway vhost — is the one shape that cannot work. A terminating proxy
strips the certificate and bao sees nginx as the client for every hive.

nginx's stream module does not terminate. `ssl_preread` reads the SNI off
the ClientHello and splices the rest of the connection through byte for
byte, so bao completes the handshake itself and authenticates the client
it actually has. That is the no-vhost rule kept, not bent.

The listener binds the bridge IP rather than every address, because bao
already holds `127.0.0.1:<port>` in the same netns and a wildcard bind
there is EADDRINUSE — nginx would fail to start, taking the gateway with
it. Nothing moves as a result: the name already resolves two ways, so a
host-side reader still goes straight to loopback and an agent goes
through the passthrough, both on one `BAO_ADDR`.

Renders only inside the store's own `deploy.bao.enable` region; a host
that runs no store grows no listener and opens no port.

Per-agent certificates and per-agent policy are separate work.

Refs #4386
This commit is contained in:
atlas 2026-09-13 23:08:38 +02:00
commit ef2dfbfb31
3 changed files with 145 additions and 2 deletions

View file

@ -15,8 +15,10 @@
# a non-HTTP protocol. It speaks HTTPS, so nginx *could* front it: **the client
# certificate IS the authentication**, and a terminating proxy strips it,
# leaving bao seeing nginx as the client for every hive in the swarm — one
# identity where there must be many. Reach is loopback plus whatever
# `deploy.bao.extraListenAddresses` names.
# identity where there must be many. Reach is loopback, whatever
# `deploy.bao.extraListenAddresses` names, and — for readers inside an agent
# container — the nginx *stream* passthrough below, which routes on the SNI
# without decrypting and so leaves the client certificate intact.
#
# ⚠️ THIS MODULE HAS NO OPINION ABOUT WHERE THE STORE'S IDENTITY COMES FROM.
# A store must not take its certificates from an authority it will itself
@ -795,6 +797,39 @@ in
# dead exporter rather than a wrong address.
bao = "127.0.0.1:${toString baoDeploy.metricsPort}/v1/sys/metrics?format=prometheus";
};
# ⚠️ The one nginx exception to this file's header, and it is one because
# it never terminates. `ssl_preread` reads the SNI off the ClientHello
# and splices the rest through byte for byte, so bao completes the
# handshake itself and sees the *client's* certificate — the exact
# property the no-vhost rule protects. A vhost would decrypt here and
# hand the store one identity for every hive in the swarm.
#
# Bridge IP, never `0.0.0.0`: the store shares this netns and already
# holds `127.0.0.1:<port>`, so a wildcard bind on that port is
# `EADDRINUSE` and nginx would not start at all. Nothing is lost to the
# split, because the name already resolves two ways — `/etc/hosts` sends
# a host-side reader straight to loopback, dnsmasq answers an agent's
# query with the bridge IP. One name, one port, one `BAO_ADDR`; only
# which netns asked decides whether the passthrough is in the path.
services.nginx.streamConfig = ''
map $ssl_preread_server_name $swarm_bao_backend {
${cfg.domain} 127.0.0.1:${toString cfg.port};
default "";
}
server {
listen ${networkCfg.bridgeIp}:${toString cfg.port};
ssl_preread on;
proxy_pass $swarm_bao_backend;
}
'';
# Firewall only: the listener above is already the bridge-IP bind this
# option's own documentation asks a host service for. Reaching the port
# still proves nothing — bao answers nothing without a client
# certificate its CA signed.
services.hyperhive.network.exposeHostPorts = [ cfg.port ];
})
(lib.mkIf (hyperhiveCfg.enable && deployCfg.bao.enable && haveServerTls) {