mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

gateway: per-vhost public flag, private listen addresses, and IPv6 listeners

Browse source
This commit is contained in:
atlas 2026-11-09 21:38:36 +01:00
commit fbbdf26a19
8 changed files with 364 additions and 25 deletions
Download patch file Download diff file Expand all files Collapse all files

27
docs/2-handbook/swarm/README.md
View file

@ -616,6 +616,33 @@ set `deploy.bao.openFirewall` without `deploy.bao.clientCaFile`. Point
`nats.<swarm.domain>` and `bao.<swarm.domain>` in your upstream DNS at the
address the other hives reach.
### Keeping a vhost off the public address
The gateway is one nginx per host, and a vhost answers on every address of the
host by default, IPv4 and IPv6. To serve some vhosts only inside a private
network (a VPN, say), list the host's addresses on that network in
`services.hyperhive.gateway.privateListenAddresses` and set `public = false`
on the vhosts that should stay inside:
```nix
services.hyperhive.gateway = {
privateListenAddresses = [ "10.100.0.1" "fd00:100::1" ];
vhosts."grafana.myswarm.example".public = false;
};
```
A `public = false` vhost listens on those addresses only. The public address
never serves it, and a request for its name there closes without a response.
Vhosts that keep the default `public = true` answer on the private addresses
too. The addresses can be IPv4 or IPv6 (`fd00::1` or `[fd00::1]`), and each
must exist on the host when nginx starts. The firewall stays your call: this
decides which vhosts nginx serves on which address, not which ports are open.
The gateway also listens on `[::]` by default, so the vhosts are reachable
over IPv6. `gateway.listenIPv6` follows `networking.enableIPv6`; turn it off
on a host whose kernel can't open an IPv6 socket. Details are in
[`gateway.md`](../../4-implementation/networking/gateway.md#listen-addresses).
### WireGuard inter-hive mesh (optional)
The peer config above uses public HTTPS for all inter-hive traffic.

19
docs/4-implementation/networking/gateway.md
View file

@ -22,7 +22,7 @@ You rarely switch it on yourself. `gateway.enable` defaults to off, and every mo
| `<queue domain>/metrics` | the queue exporter's `/metrics` on loopback, behind an authelia `auth_request`; every other path, `/` included, returns `403`, so `/varz` and `/connz` are never reachable | `swarm-nats.nix`, the queue's host |
| `grafana.<swarm>`, `metrics.<swarm>`, `logs.<swarm>`, `otel.<swarm>`, `bao.<swarm>` | the matching swarm service | that service's module → [`2-handbook/swarm/services.md`](../../2-handbook/swarm/services.md) |
A service module declares its vhost as `services.hyperhive.gateway.vhosts.<name>`: the locations, `forceSSL` and any `extraConfig` it needs. The gateway renders it into `services.nginx.virtualHosts.<name>` and adds the `listen` set, the certificate for that name and the [security headers](#security-headers) (ahead of the vhost's own `extraConfig`), and drops `addSSL` from a vhost that sets `forceSSL`. The eval fails for a vhost that sets `listen`, `addSSL`, `onlySSL`, `enableACME`, `useACMEHost`, `sslCertificate` or `sslCertificateKey` itself, and for any vhost declared while `gateway.enable` is false. A name with a certificate from outside the gateway's leaves (the queue's) sets `certificate = { cert; key; }` instead. `hive-gateway/vhosts.nix` declares the gateway's own catch-all (`_`) the same way, `hive-c0re/vhosts.nix` the dashboard and the hive domain, and `glue-matrix-hive-discovery.nix` adds the matrix redirect to the hive domain where the homeserver runs, and discovery there when the hive domain is the pinned server name. The catch-all sets `securityHeaders = false` because it sends no response, and the gateway renders a vhost's `default` at `mkDefault` priority.
A service module declares its vhost as `services.hyperhive.gateway.vhosts.<name>`: the locations, `forceSSL` and any `extraConfig` it needs. The gateway renders it into `services.nginx.virtualHosts.<name>` and adds the `listen` set ([listen addresses](#listen-addresses)), the certificate for that name and the [security headers](#security-headers) (ahead of the vhost's own `extraConfig`), and drops `addSSL` from a vhost that sets `forceSSL`. The eval fails for a vhost that sets `listen`, `addSSL`, `onlySSL`, `enableACME`, `useACMEHost`, `sslCertificate` or `sslCertificateKey` itself, and for any vhost declared while `gateway.enable` is false. A name with a certificate from outside the gateway's leaves (the queue's) sets `certificate = { cert; key; }` instead. `hive-gateway/vhosts.nix` declares the gateway's own catch-all (`_`) the same way, `hive-c0re/vhosts.nix` the dashboard and the hive domain, and `glue-matrix-hive-discovery.nix` adds the matrix redirect to the hive domain where the homeserver runs, and discovery there when the hive domain is the pinned server name. The catch-all sets `securityHeaders = false` because it sends no response, and the gateway renders a vhost's `default` at `mkDefault` priority.
**The hive's own vhosts**: the dashboard's, named `<name>.hives.<swarm>` (`deploy.hive-controller.dashboardDomain`), and one named for the hive domain. They exist only on a host that runs a hive (`deploy.hive-controller.enable`); a host without a hive has none of these rows:
@ -124,6 +124,23 @@ The name sits under the swarm domain so authelia's session cookie reaches it, wh
**Certificate:** in self-signed mode the name is a SAN on the hive's own leaf, issued through its own store role, so it needs nothing extra. With `tls.acme` it needs a public DNS record like any other vhost. With `tls.certDir` your certificate must name it; a `*.<swarm>` wildcard doesn't cover `<name>.hives.<swarm>`.
## Listen addresses
Every vhost listens on `port` and `httpsPort` (80 and 443 by default). On which addresses depends on the vhost's `public` flag:
| vhost | listens on |
|---|---|
| `public = true` (default) | `0.0.0.0`, `[::]` (unless `listenIPv6` is off), and each of `privateListenAddresses` |
| `public = false` | each of `privateListenAddresses` only |
`gateway.privateListenAddresses` holds addresses of this host, IPv4 or IPv6, written bare or bracketed (`fd00::1`, `[fd00::1]`). The list is empty by default, so every vhost is public, on `0.0.0.0` and `[::]`. The eval refuses a vhost with `public = false` while the list is empty, and refuses a wildcard (`0.0.0.0`, `::`) in the list, since that would open the private vhosts on the public address. Each address must exist on the host when nginx starts (a WireGuard interface's address, say), or nginx fails to bind.
`gateway.listenIPv6` adds the `[::]` listens. It defaults to `networking.enableIPv6`, so a host that has IPv6 turned off keeps listening on `0.0.0.0` only. nginx opens the `[::]` socket IPv6-only (its default since 1.3.4), so it sits next to `0.0.0.0` on the same port without a conflict, and the IPv4 listeners keep working as before. `openFirewall` opens the ports for both address families.
**Why public vhosts also list the private addresses.** nginx chooses the server set from the address a connection arrived on, before it looks at `Host`. Once any vhost names `10.100.0.1:443`, that address has a server set of its own, and a vhost that lists only the wildcard doesn't belong to it. The vhosts that stay public, and the catch-all (`_`, which carries `default_server`), therefore list the private addresses too. Without that, an unknown or public `Host` on a private address would land on an arbitrary private vhost.
**Why the public address never reaches a private vhost.** nginx matches a connection to the public address against the wildcard listens only, and no `public = false` vhost listens there. With a wildcard listen present, nginx opens just the wildcard sockets and picks the server set from the connection's local address, so you need no `bind` parameter. A request for a private name on the public address gets the catch-all's `444`, the same as any unknown `Host`.
## Firewall posture (host-level)
`services.hyperhive.gateway.openFirewall = true` opens `port` and `httpsPort` on the host firewall — both, since the gateway always serves TLS. It defaults to off; set it for any reach from outside the host.

31
nix/host-modules/hive-gateway/declared-vhosts.nix
View file

@ -1,7 +1,7 @@
# Renders `services.hyperhive.gateway.vhosts` into
# `services.nginx.virtualHosts`: each declared vhost keeps what its module
# wrote and gets the gateway's `listen`, the TLS attributes for its name and
# the security headers.
# wrote and gets the gateway's `listen` (the private one when it sets
# `public = false`), the TLS attributes for its name and the security headers.
#
# Its own module rather than part of ./default.nix because the
# "declared while the gateway is off" assertion has to hold when
@ -9,6 +9,7 @@
{ lib, config, ... }:
let
cfg = config.services.hyperhive.gateway;
listen = import ./listen.nix { inherit lib; };
# What the gateway owns on a vhost. A declared vhost naming one of these
# fails the eval.
@ -26,12 +27,16 @@ let
name: map (attr: "${name}.${attr}") (lib.filter (a: cfg.vhosts.${name} ? ${a}) gatewayOwned)
) (lib.attrNames cfg.vhosts);
privateOnly = lib.filter (name: !cfg.vhosts.${name}.public) (lib.attrNames cfg.vhosts);
wildcardPrivate = lib.filter listen.isWildcard cfg.privateListenAddresses;
render =
name: declared:
let
own = removeAttrs declared [
"_module"
"certificate"
"public"
"securityHeaders"
];
base =
@ -48,7 +53,7 @@ let
tls
// own
// {
listen = cfg.lib.listen;
listen = if declared.public then cfg.lib.listen else cfg.lib.privateListen;
extraConfig =
lib.optionalString declared.securityHeaders cfg.lib.securityHeaders + (own.extraConfig or "");
}
@ -83,6 +88,26 @@ in
`certificate` on the vhost.
'';
}
{
assertion = privateOnly == [ ] || cfg.privateListenAddresses != [ ];
message = ''
services.hyperhive.gateway.vhosts sets `public = false` on
${lib.concatStringsSep ", " privateOnly}
while services.hyperhive.gateway.privateListenAddresses is empty,
so these vhosts would listen nowhere. Set the addresses they
should answer on, or drop `public = false`.
'';
}
{
assertion = wildcardPrivate == [ ];
message = ''
services.hyperhive.gateway.privateListenAddresses holds
${lib.concatStringsSep ", " wildcardPrivate}, which is every address
of the host, not a private one: the vhosts with `public = false`
would answer on the public address too. List the host's private
addresses (a WireGuard interface's, say).
'';
}
];
}
(lib.mkIf cfg.enable {

52
nix/host-modules/hive-gateway/listen.nix Normal file
View file

@ -0,0 +1,52 @@
# The gateway's listen sets, as pure functions of `lib` and the gateway's
# port and address settings.
#
# nginx picks the server set for a connection by the address the connection
# arrived on: a server that lists only `10.0.0.1:443` never sees a connection
# to another address, and a server that lists only the wildcard never sees one
# to `10.0.0.1:443` once some server lists that address. So a vhost reachable
# on a private address must list it explicitly, and so must every vhost that
# should keep answering there (including the `default_server`, or an unknown
# `Host` on the private address lands on an arbitrary private vhost).
{ lib }:
rec {
# nginx wants an IPv6 literal in brackets (`[fd00::1]:443`). Accept the bare
# spelling too.
bracket = addr: if lib.hasInfix ":" addr && !lib.hasPrefix "[" addr then "[${addr}]" else addr;
# The spellings of "every address". Listing one as a private address would
# open the private vhosts on the public one.
wildcards = [
"0.0.0.0"
"[::]"
"[::0]"
"*"
];
isWildcard = addr: lib.elem (bracket addr) wildcards;
# One address, both ports: plain http on `port`, TLS on `httpsPort`.
pair = cfg: addr: [
{
inherit addr;
port = cfg.port;
}
{
inherit addr;
port = cfg.httpsPort;
ssl = true;
}
];
# `privateListenAddresses`, bracketed and deduplicated: nginx refuses a
# duplicate `listen`.
privateAddrs = cfg: lib.unique (map bracket cfg.privateListenAddresses);
# The listens of a `public = false` vhost.
private = cfg: lib.concatMap (pair cfg) (privateAddrs cfg);
# The listens of a `public = true` vhost: the wildcard (IPv6 as well when
# `listenIPv6`), plus the private addresses, which each have a server set of
# their own.
public =
cfg: lib.concatMap (pair cfg) ([ "0.0.0.0" ] ++ lib.optional cfg.listenIPv6 "[::]") ++ private cfg;
}

76
nix/host-modules/hive-gateway/options.nix
View file

@ -145,6 +145,16 @@ in
request with a response, such as the catch-all.
'';
};
options.public = lib.mkOption {
type = lib.types.bool;
default = true;
description = ''
Whether this vhost answers on every address the gateway listens
on. `false` binds it to `privateListenAddresses` only, so the
public address never serves it (see that option). Needs a
non-empty `privateListenAddresses`.
'';
};
options.certificate = lib.mkOption {
type = lib.types.nullOr (
lib.types.submodule {
@ -177,7 +187,9 @@ in
config — and the gateway renders it into
`services.nginx.virtualHosts.<name>` with the parts it owns:
- `listen`, the gateway's port pair;
- `listen`, the gateway's port pair on its addresses
(`gateway.lib.listen`, or `gateway.lib.privateListen` for a vhost
that sets `public = false`);
- the TLS attributes for that name (`gateway.lib.tlsFor`, or
`certificate` below), with `addSSL` dropped when the vhost sets
`forceSSL`;
@ -203,12 +215,23 @@ in
internal = true;
readOnly = true;
description = ''
Read-only: the `listen` set every vhost in front of this
gateway shares (plain http on `port`, TLS on `httpsPort`).
Applied to every vhost declared under `vhosts`, and read by
the gateway's own vhosts, so no vhost binds a different
set — one that did would be reachable on a port the gateway
does not consider its own.
Read-only: the `listen` set of a vhost that answers on every
address (plain http on `port`, TLS on `httpsPort`; on `0.0.0.0`,
on `[::]` when `listenIPv6` is set, and on each of
`privateListenAddresses`). Applied to every vhost declared under
`vhosts` that keeps `public = true`, and read by the gateway's own
vhosts, so no vhost binds a different set — one that did would be
reachable on a port the gateway does not consider its own.
'';
};
privateListen = lib.mkOption {
type = lib.types.listOf (lib.types.attrsOf lib.types.raw);
internal = true;
readOnly = true;
description = ''
Read-only: the `listen` set of a vhost with `public = false`: the
same port pair, on `privateListenAddresses` only.
'';
};
@ -319,6 +342,45 @@ in
'';
};
privateListenAddresses = lib.mkOption {
type = lib.types.listOf lib.types.str;
default = [ ];
example = [
"10.100.0.1"
"fd00:100::1"
];
description = ''
Addresses of this host, IPv4 or IPv6 (bare `fd00::1` or bracketed),
that the vhosts with `public = false` listen on, on `port` and
`httpsPort`. Not the wildcard: `0.0.0.0` and `::` are refused,
since they would open those vhosts on the public address too.
The vhosts that stay public listen on these addresses as well as on
the wildcard, so a client that reaches the host by a private address
still gets them; and so does the catch-all, so an unknown `Host` on a
private address closes the connection rather than being served by
a private vhost.
The addresses must exist on the host when nginx starts (a
WireGuard interface's address, say), or nginx fails to bind. Empty
by default: every vhost is public, as before.
'';
};
listenIPv6 = lib.mkOption {
type = lib.types.bool;
default = config.networking.enableIPv6;
defaultText = lib.literalExpression "config.networking.enableIPv6";
description = ''
Listen on `[::]` as well as `0.0.0.0`, on `port` and `httpsPort`.
The IPv6 listener takes IPv6 connections only; the `0.0.0.0` one
keeps the IPv4 ones. Follows `networking.enableIPv6`; set it to
`false` on a host whose kernel cannot open an IPv6 socket.
`openFirewall` opens the ports for both address families.
'';
};
statusPort = lib.mkOption {
type = lib.types.port;
default = 8085;

24
nix/host-modules/hive-gateway/vhost-lib.nix
View file

@ -11,7 +11,8 @@
#
# Pure function of the gateway's config + resolved cert paths; returns
# an attrset, evaluates no options itself. Everything here is *gateway*
# knowledge — which port pair to bind, which issuer covers a name.
# knowledge — which addresses and port pair to bind, which issuer covers a
# name.
{
lib,
cfg, # services.hyperhive.gateway
@ -24,6 +25,8 @@
caBundle, # hive TLS trust bundle on the host
}:
let
listenSets = import ./listen.nix { inherit lib; };
# nixos `services.nginx.virtualHosts.<name>` ssl attrs for a vhost
# covered by the hive's own cert. For ACME mode: `enableACME` +
# `addSSL` — NixOS's ACME integration manages the cert lifecycle and
@ -52,18 +55,13 @@ in
# floor when neither `tls.certDir` nor ACME is set, so there is no
# http-only mode. Listen addresses every vhost shares — plain http
# on `cfg.port` plus TLS on `cfg.httpsPort`. See `docs/4-implementation/networking/gateway.md`
# ("TLS modes").
listen = [
{
addr = "0.0.0.0";
port = cfg.port;
}
{
addr = "0.0.0.0";
port = cfg.httpsPort;
ssl = true;
}
];
# ("TLS modes", "Listen addresses").
#
# `listen` is for a vhost that answers everywhere: the wildcard plus the
# private addresses. `privateListen` is for one that answers only on the
# private addresses.
listen = listenSets.public cfg;
privateListen = listenSets.private cfg;
# TLS attrs for one vhost, by name. A swarm service's name may sit
# outside this hive's domain — and then the hive leaf's role

148
nix/module-eval/core-toggle.nix
View file

@ -75,11 +75,43 @@ let
};
# The controller behind a gateway on a non-default TLS port, with the
# firewall opened. Its collector URL and its listener are read off it.
# firewall opened, a private IPv4 and IPv6 address, and one vhost kept off the
# public address. Its collector URL and its listeners are read off it.
controllerAltPort = hive {
deploy.swarm-controller.enable = true;
gateway.httpsPort = 8443;
gateway.openFirewall = true;
gateway.privateListenAddresses = [
"10.100.0.1"
"fd00:100::1"
];
gateway.vhosts."internal.t.local".public = false;
};
# A private address that is the wildcard, beside a vhost that asks to be
# private: the listen set is well-formed, the privacy it promises is not.
privateWildcard = hive {
gateway.privateListenAddresses = [ "0.0.0.0" ];
gateway.vhosts."internal.t.local".public = false;
};
# Only its `networking.enableIPv6` default is read.
noIPv6 = hiveWith [ { networking.enableIPv6 = false; } ] { };
gatewayListen = import ../host-modules/hive-gateway/listen.nix { inherit lib; };
# nginx fills defaults into each `listen` entry; compare the fields the
# gateway sets.
listenFields = map (l: {
inherit (l) addr port;
ssl = l.ssl or false;
});
httpAt = addr: port: {
inherit addr port;
ssl = false;
};
httpsAt = addr: port: {
inherit addr port;
ssl = true;
};
# A container the operator runs that hyperhive doesn't own, next to one it
@ -2176,6 +2208,120 @@ let
&& lib.length (checks "swarm-bao") == 1
&& checks "operator-box" == [ ];
}
{
# Presence controls: the IPv6 arm is on in this fixture, and the private
# list is empty, so what is absent below is absent by configuration.
name = "a gateway with no private addresses listens on both ports of 0.0.0.0 and [::] and nowhere else";
ok =
let
gw = allLocal.services.hyperhive.gateway;
l = allLocal.services.nginx.virtualHosts."_".listen;
in
gw.listenIPv6
&& gw.privateListenAddresses == [ ]
&&
listenFields l == [
(httpAt "0.0.0.0" 80)
(httpsAt "0.0.0.0" 443)
(httpAt "[::]" 80)
(httpsAt "[::]" 443)
]
&& listenFields gw.lib.listen == listenFields l
&& gw.lib.privateListen == [ ];
}
{
# `listenIPv6` is the escape for a host whose kernel cannot open an IPv6
# socket, and follows `networking.enableIPv6` unless set.
name = "the IPv6 listener can be turned off and follows networking.enableIPv6";
ok =
let
off = {
port = 80;
httpsPort = 443;
listenIPv6 = false;
privateListenAddresses = [ ];
};
in
listenFields (gatewayListen.public off) == [
(httpAt "0.0.0.0" 80)
(httpsAt "0.0.0.0" 443)
]
&& !noIPv6.services.hyperhive.gateway.listenIPv6
&& allLocal.services.hyperhive.gateway.listenIPv6;
}
{
# An IPv6 literal is bracketed whichever way the operator wrote it, and
# listed once: nginx refuses a duplicate `listen`. The wildcard test is
# paired with addresses that are not wildcards.
name = "private addresses render IPv6 in brackets, once, and the wildcards are recognised";
ok =
let
cfg = {
port = 80;
httpsPort = 443;
listenIPv6 = true;
privateListenAddresses = [
"10.100.0.1"
"fd00:100::1"
"[fd00:100::1]"
];
};
in
listenFields (gatewayListen.private cfg) == [
(httpAt "10.100.0.1" 80)
(httpsAt "10.100.0.1" 443)
(httpAt "[fd00:100::1]" 80)
(httpsAt "[fd00:100::1]" 443)
]
&& lib.all gatewayListen.isWildcard [
"0.0.0.0"
"::"
"[::]"
"*"
]
&& !(lib.any gatewayListen.isWildcard cfg.privateListenAddresses);
}
{
# The private vhost binds the private addresses only; the catch-all and
# every vhost that stays public bind the wildcards AND the private
# addresses, because nginx gives a private address a server set of its
# own. The catch-all arm is the presence control for the wildcard check.
name = "a vhost with public = false listens on the private addresses only, the catch-all on those and the wildcards";
ok =
let
vh = controllerAltPort.services.nginx.virtualHosts;
private = [
(httpAt "10.100.0.1" 80)
(httpsAt "10.100.0.1" 8443)
(httpAt "[fd00:100::1]" 80)
(httpsAt "[fd00:100::1]" 8443)
];
in
listenFields vh."internal.t.local".listen == private
&& !(lib.any (l: l.addr == "0.0.0.0" || l.addr == "[::]") vh."internal.t.local".listen)
&&
listenFields vh."_".listen == [
(httpAt "0.0.0.0" 80)
(httpsAt "0.0.0.0" 8443)
(httpAt "[::]" 80)
(httpsAt "[::]" 8443)
]
++ private
&& vh."_".default;
}
{
# Both arms: the wildcard refusal fires on the wildcard, and the
# "needs an address" refusal stays quiet there because the list is not
# empty.
name = "a wildcard private address is refused";
ok =
let
fired = lib.filter (a: !a.assertion) privateWildcard.assertions;
mentions = s: lib.any (a: lib.hasInfix s a.message) fired;
in
mentions "privateListenAddresses holds"
&& !(mentions "while services.hyperhive.gateway.privateListenAddresses is empty");
}
]
# Starting a unit that stays `active (exited)` is a no-op, so a timer aimed
# at one fires and fetches nothing.

12
nix/module-eval/grafana.nix
View file

@ -91,6 +91,8 @@ let
swarm.authelia.url = "https://auth.example.invalid";
# Declares a vhost that binds its own `listen`, which the gateway refuses.
gateway.vhosts."own-listen.t.local".listen = [ ];
# Asks to be private on a gateway with no private address to be private on.
gateway.vhosts."internal.t.local".public = false;
};
# Did ./host-modules/swarm-grafana.nix refuse this host, and for which of its
@ -1081,6 +1083,16 @@ let
&& lib.hasInfix "own-listen.t.local.listen" a.message
) grafanaNoIdentity.assertions;
}
{
# A private vhost with nowhere to listen would be silently unserved.
name = "a vhost with public = false is refused while there is no private address";
ok = lib.any (
a:
!a.assertion
&& lib.hasInfix "privateListenAddresses is empty" a.message
&& lib.hasInfix "internal.t.local" a.message
) grafanaNoIdentity.assertions;
}
];
in
runGroup "grafana" cases