vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(3088): move the gateway's nginx + dnsmasq onto the host

Browse source 
The gateway's nginx + dnsmasq no longer run in their own nspawn container.
`nix/host-modules/hive-gateway/default.nix` loses the
`containers.hive-gateway` wrapper and everything that existed only to punch
holes in it: `privateNetwork = false`, `CAP_NET_ADMIN`, five bind mounts,
its own `stateVersion`, `networking.firewall.enable = false`,
`networking.resolvconf.enable = false`, and the `hive-gateway-resolv`
path+service pair. 465 -> 303 lines.

The container never bought isolation here. It shared the host netns by
necessity — nginx binds the host's :80/:443, dnsmasq answers on the bridge —
so each of those settings was undoing a boundary the gateway could not
afford in the first place.

Four things made it more than a deletion, none of them visible in the nix
diff:

- The self-signed cert service also imports the hive CA leaf, so removing it
  with the container would have left nginx naming a missing cert file, which
  it refuses to load at all.
- The nginx reload is a hive-priv verb. It still needs root, but no longer
  for the reason its doc gave, and `--machine=` was both transport and
  scope — so the unit name is now hard-coded in the helper as the
  containment.
- The lifecycle verb named a container that stops existing.
- `journalctl -M hive-gateway` had no machine to enter.

Per the operator's ruling, the operator verb keeps working and agents lose
it. `InfraContainer` answered three questions that used to share an answer;
it now splits into `name()` (identity), `target()` (Container vs HostUnit),
`service_unit()` (the systemd unit), and `agent_restartable()`, which the
MCP restart path checks before the capability so the refusal cannot read as
"ask for infra_admin". `SIBLING_CONTAINERS` drops the gateway — it gates the
requests that name a container as a string — while `FromStr` still accepts
it, because that answers what a name is, not who may act on it. The
dashboard's gateway journal reads host journald filtered to `nginx.service`.

Prose was corrected where it only named a location, and re-argued where the
container was doing security work: a `0666` per-agent socket was safe
because only the gateway container had the directory bind-mounted. There is
no mount now, so the directory permissions are the whole of the access
control — the constraint holds, its mechanism doesn't.

Gate: nix fmt / clippy --all-targets -D warnings / cargo test all clean (710
tests); hivectl-cli.md regenerated from the clap tree. The nix eval was run
in both TLS shapes at this commit: every delta in the rendered
virtualHosts is one of the three intended path moves, dnsmasq settings are
byte-identical, and the absence probe flips true -> false with bindMounts
emptied.
This commit is contained in:
atlas 2026-08-11 18:00:27 +02:00
commit 07852cabc1
34 changed files with 704 additions and 618 deletions
Download patch file Download diff file Expand all files Collapse all files

531
nix/host-modules/hive-gateway/default.nix
View file

@ -1,7 +1,10 @@
# Single nginx in front of every hyperhive web surface — dashboard,
# per-agent UIs (sub-path), forge + matrix (sub-domain), .well-known
# delegations — plus the hive-internal dnsmasq resolver, co-located in
# the same `hive-gateway` container (shared host netns, state-free).
# delegations — plus the hive-internal dnsmasq resolver. Both run on the
# HOST, next to hive-c0re. They used to live in a `hive-gateway`
# container that shared the host netns anyway, so the boundary bought no
# network isolation and cost a resolv.conf sync, a reload that had to
# cross the machine bus, and four bind mounts.
# Full vhost map + discovery flow + design rationale in
# `docs/gateway.md`. Layout: ./options.nix (option declarations),
# ./vhosts.nix (the nginx virtual-host tree), ./error-pages.nix
@ -42,6 +45,43 @@ let
# so the gateway always terminates TLS.
# `cfg.useSelfSigned` (options.nix) is the derived single source of truth.
useSelfSigned = cfg.useSelfSigned;
# nginx's own state dir. Kept at the historical `/var/lib/hive-gateway`
# path rather than renamed with the move: it holds the imported leaf
# across reboots, and renaming it would strand every existing hive's
# certs for no gain.
tlsDir = "/var/lib/hive-gateway/tls";
# TLS cert + key paths.
# - self-signed (default): the hive-CA-signed leaf, imported into the
# state dir by `hive-gateway-self-signed-cert` below.
# - tls.certDir set: the operator's own cert dir, read directly.
tlsCert =
if cfg.tls.certDir != null then "${cfg.tls.certDir}/${cfg.tls.certName}" else "${tlsDir}/cert.pem";
tlsKey =
if cfg.tls.certDir != null then "${cfg.tls.certDir}/${cfg.tls.keyName}" else "${tlsDir}/key.pem";
# The swarm-services pair, used only by the vhosts whose names this
# hive's CA cannot sign. Self-signed mode only: with an operator cert
# or ACME the operator owns every name and there is no second issuer.
svcCert = "${tlsDir}/swarm-services.pem";
svcKey = "${tlsDir}/swarm-services-key.pem";
nginxTree = import ./vhosts.nix {
inherit
lib
cfg
forgeCfg
matrixCfg
hyperhiveDomain
dashboardDist
swaggerUiTheme
tlsCert
tlsKey
svcCert
svcKey
swarmServiceDomains
;
errorPages = import ./error-pages.nix { inherit pkgs; };
};
in
{
imports = [ ./options.nix ];
@ -65,10 +105,12 @@ in
}
];
# Ensure bind-mount sources exist at host boot before the gateway
# container's first start. nspawn would auto-create missing dirs;
# tmpfiles rules make the intent explicit and cover the fresh-boot
# window before c0re has run.
# Ensure the gateway state dirs exist at host boot, before anything
# reads or writes them. They used to double as bind-mount sources
# for the container (nspawn would auto-create a missing one); the
# rules stay because they still cover the fresh-boot window before
# c0re has run, and they pin owner + mode rather than leaving it to
# whoever creates the path first.
#
# /run/hive-agent — per-agent UDS socket dir, written by c0re's
# set_nspawn_flags when agents start. Owned by `hive-core` (the
@ -95,349 +137,170 @@ in
"f /var/lib/hyperhive/gateway/gateway.htpasswd 0644 root root - -"
];
# Keep the gateway's `/etc/resolv.conf` in step with the host's.
# ⚠️ REMOVED WITH THE CONTAINER, and each one was a workaround for the
# boundary rather than a thing nginx or dnsmasq needed:
#
# nixos-container does `cp --remove-destination /etc/resolv.conf
# "$root/etc/resolv.conf"` in its start script, and nspawn's
# `--resolv-conf=auto` copies rather than binds for a writable,
# host-netns container like this one. Both are one-shot: systemd-nspawn(1)
# says outright that "no further propagation of configuration is
# generally done after the one-time early initialization (this is
# because the file is usually updated through copying and renaming)".
# - `privateNetwork = false` — the container already shared the host
# netns, which is why nginx bound host ports and `localhost`
# upstreams reached hive-c0re. On the host that is simply true.
# - `additionalCapabilities = [ "CAP_NET_ADMIN" ]` — dnsmasq refuses
# to start with a `dhcp-range` unless it holds NET_ADMIN, and
# nspawn's bounding set dropped it for a host-netns container.
# Host root has it.
# - `networking.firewall.enable = false` — a container sharing the
# host netns would run *its* firewall.service against the HOST
# ruleset, flushing nixos-fw and deleting the nixos-nat-* chains on
# every boot. With one machine there is one firewall (below).
# - `networking.resolvconf.enable = false` + the `hive-gateway-resolv`
# path/service pair — the container's /etc/resolv.conf was a
# one-shot copy frozen at start, so a host network change left
# dnsmasq forwarding to a resolver that was gone. The whole
# watch-copy-reload machine existed to bridge two files. There is
# now one.
# - three bind mounts — /run/hive-agent, /run/hive-state, and either
# /run/hive-tls (operator cert) or /run/hive-ca (self-signed);
# those last two are mutually exclusive mkIfs, so it was never
# four. All plain host paths now.
#
# So the gateway's copy is frozen at container start. dnsmasq has no
# explicit upstream (see ./dnsmasq.nix) and follows that file, which
# means a host network change — new router, new DHCP lease, laptop
# moving between networks — leaves dnsmasq forwarding to a resolver
# that is gone, and every non-hive lookup from every agent hangs. The
# agents' own resolvers point at the static bridge IP and never go
# stale, which is exactly why the symptom presents as "the gateway
# needs a kick".
#
# Hence: watch on the host, push into the container, reload dnsmasq.
# `reload` is `kill -HUP $MAINPID` (the upstream dnsmasq unit's
# ExecReload), so dnsmasq re-reads its upstream list and drops its
# cache without severing anything — nginx never notices. Why this has
# to run host-side, and why a file bind-mount is worse than the copy:
# `docs/network.md::Resolver behaviour`.
systemd.paths.hive-gateway-resolv = {
description = "Watch the host's /etc/resolv.conf for the hive-gateway container";
wantedBy = [ "multi-user.target" ];
# Arm the watch before anything configures the network, so the very
# first DHCP-driven resolv.conf write of the boot is caught too —
# that's the "gateway came up while DHCP was still settling" case.
# Inert if nothing pulls network-pre.target into the transaction.
before = [ "network-pre.target" ];
pathConfig = {
# PathChanged also watches the parent directory, so openresolv's
# atomic rename-over lands as IN_MOVED_TO on /etc and triggers —
# a watch on the inode alone would die with the replaced file.
PathChanged = "/etc/resolv.conf";
Unit = "hive-gateway-resolv.service";
};
# See `docs/network.md::Resolver behaviour` for the resolver history.
# ACME (Let's Encrypt) integration. nginx vhosts set
# `enableACME = true` via the vhost builder; this provides the
# shared ACME config (acceptTerms + email).
security.acme = lib.mkIf cfg.tls.acme.enable {
acceptTerms = true;
defaults.email = cfg.tls.acme.email;
};
systemd.services.hive-gateway-resolv = {
description = "Sync the host's resolvers into hive-gateway and reload dnsmasq";
# Also run once per gateway start, to catch a host resolver change
# that happened while the container was down. Deliberately NOT
# ordered after network-online.target: pulling that target in on
# every resolv.conf change risks blocking the sync behind a
# wait-online timeout on hosts where nothing else reaches it. The
# boot race is closed by arming the path unit early instead.
wantedBy = [ "container@hive-gateway.service" ];
after = [ "container@hive-gateway.service" ];
path = [
pkgs.systemd
pkgs.coreutils
pkgs.gnugrep
];
# Import the hive-CA leaf into nginx's state dir before nginx starts.
#
# 🚨 THIS LOOKS LIKE A LEFTOVER OF THE CONTAINER AND IS NOT — do not
# "simplify" it into pointing nginx at the CA dir. It does TWO jobs:
#
# (1) It re-modes the leaf. `hive-tls-ca` writes the key 0600
# root:root; nginx's pre-start `nginx -t` runs as the nginx
# *user*, so a 0600 key fails the config test with
# `BIO_new_file() … Permission denied` and blocks the unit —
# hence the 0640 root:nginx copy below. That is a file-mode fact,
# not a namespace one, and it did not go away with the container.
# (2) It guarantees that **every cert path the nginx config names
# exists** — which is what the swarm-services fallback at the
# bottom of the script is for. nginx refuses to load a config
# naming a missing cert file, so a leaf that never issues takes
# the whole gateway down rather than one vhost; that has already
# happened once and it took the forge, dashboard and matrix with
# it. Removing this unit re-creates it exactly.
#
# nginx
# `Requires=` this via `requiredBy`, so it refuses to start until
# the copy succeeds. ALWAYS runs (no ConditionPathExists) and is
# idempotent — necessary to reconcile broken state from prior
# failed boots (a 0700 dir from a stale UMask, a truncated copy
# from an interrupted oneshot, etc.). The leaf covers the bare
# hive domain plus `forge.`, `matrix.` and `*.${hyperhiveDomain}`
# so all sub-domains validate under the same cert + the hive CA.
# See `docs/gateway.md` ("Self-signed TLS").
systemd.services.hive-gateway-self-signed-cert = lib.mkIf useSelfSigned {
description = "Import host-generated TLS leaf for hive-gateway";
wantedBy = [ "multi-user.target" ];
before = [ "nginx.service" ];
requiredBy = [ "nginx.service" ];
serviceConfig = {
Type = "oneshot";
SyslogIdentifier = "hive-gateway-resolv";
RemainAfterExit = true;
# Pin the journal identity (else it's the `script` store-path wrapper).
SyslogIdentifier = "hive-gateway-self-signed-cert";
};
path = [ pkgs.coreutils ];
script = ''
set -euo pipefail
src=/etc/resolv.conf
# Marker lives on tmpfs, so the first sync after every host boot
# always goes through rather than trusting a stale comparison.
marker=/run/hive-gateway/resolv.synced
set -eu
mkdir -p ${tlsDir}
# 0755 on BOTH the cert dir and its parent so the nginx
# user can traverse the full path. The parent
# `/var/lib/hive-gateway` lands at 0700 by default (systemd
# StateDirectory / mkdir umask depending on which service
# created it first), which on its own blocks traversal.
# Re-applied every boot in case a prior run left a tighter
# mode behind.
chmod 0755 ${builtins.dirOf tlsDir}
chmod 0755 ${tlsDir}
# Copy the host leaf in. `install` writes atomically with the
# target mode; run as root (container root == host root,
# privateUsers=false) so the 0600 root:root host key is
# readable. Key ends up root:nginx 0640 so nginx-pre-start
# (which runs `nginx -t` as the nginx user, not root) can
# read it — a 0600 root:root key passes the master load but
# fails the pre-start config test with `BIO_new_file() …
# Permission denied`, blocking the unit. Cert is world-read.
install -m 0644 ${config.services.hyperhive.tls.stateDir}/gateway.pem ${tlsCert}
install -m 0640 -g nginx ${config.services.hyperhive.tls.stateDir}/gateway-key.pem ${tlsKey}
# No nameserver line means either a mid-rewrite snapshot or a host
# with no DNS at all. In both cases the gateway's existing copy is
# the best information available — keep it and wait for the next
# event, instead of pushing a file that resolves nothing.
if ! grep -q '^[[:space:]]*nameserver[[:space:]]' "$src"; then
echo "host resolv.conf has no nameserver line — keeping the gateway's current copy"
exit 0
# The swarm-services leaf, when this host issues one. It is
# a separate pair rather than more SANs on the one above
# because no hive CA can sign these names — each is
# constrained to its own hive's domain and the service
# names are siblings of it.
#
# Absent is a normal state, not a failure: the leaf exists
# only where the swarm CA is autoconfigured, and issuance
# can also fail on a host that wants one.
#
# ⚠️ When it is absent the HIVE leaf goes to this path
# anyway, and that fallback is load-bearing rather than
# tidy. nginx refuses to load a config naming a cert file
# that does not exist — `cannot load certificate … no such
# file` fails the pre-start test, so the vhost does not
# degrade, the ENTIRE gateway dies and takes the forge, the
# dashboard and matrix with it. Serving the hive leaf on a
# swarm-service name is a name mismatch: browsers warn,
# strict clients refuse, everything else keeps working, and
# the operator gets a bad cert instead of no hive.
#
# Measured, not theorised: this exact path took pr1ma's
# gateway down when the services sub-CA failed to issue.
if [ -s ${config.services.hyperhive.tls.stateDir}/swarm-services.pem ]; then
install -m 0644 ${config.services.hyperhive.tls.stateDir}/swarm-services.pem ${svcCert}
install -m 0640 -g nginx ${config.services.hyperhive.tls.stateDir}/swarm-services-key.pem ${svcKey}
else
echo "no swarm-services leaf — serving the hive leaf on those names (mismatch, not an outage)" >&2
install -m 0644 ${config.services.hyperhive.tls.stateDir}/gateway.pem ${svcCert}
install -m 0640 -g nginx ${config.services.hyperhive.tls.stateDir}/gateway-key.pem ${svcKey}
fi
if [ -e "$marker" ] && cmp -s "$src" "$marker"; then
echo "host resolvers unchanged since last sync — nothing to do"
exit 0
fi
# A stopped gateway needs no push: its next start copies the
# current host file itself.
if ! systemctl is-active --quiet container@hive-gateway.service; then
echo "hive-gateway not running — its next start copies the current file itself"
exit 0
fi
# `machinectl copy-to` writes through the container's own mount
# namespace, so this stays correct regardless of how the container
# assembles /etc (e.g. if system.etc.overlay is ever turned on) —
# unlike poking at the rootfs path from the host side.
machinectl copy-to hive-gateway "$src" /etc/resolv.conf --force
# Not fatal: dnsmasq polls resolv.conf for mtime changes on its own,
# so a failed reload costs a second of staleness, not correctness.
if ! systemctl -M hive-gateway reload dnsmasq.service; then
echo "dnsmasq reload failed (not up yet?) — copy is in place, its own poll will pick it up"
fi
mkdir -p "$(dirname "$marker")"
install -m 0644 "$src" "$marker"
echo "synced host resolvers into hive-gateway and reloaded dnsmasq"
'';
};
containers.hive-gateway = {
autoStart = true;
ephemeral = false;
# Share host netns — nginx then binds host-level ports directly,
# `localhost` upstream resolution reaches hive-c0re without any
# port-forward dance, and the firewall config below is the only
# layer that matters.
privateNetwork = false;
# dnsmasq refuses to start once a dhcp-range is configured unless it
# holds CAP_NET_ADMIN (DNS-only mode doesn't need it). Private-network
# containers retain NET_ADMIN implicitly, but this container shares the
# host netns (above), so nspawn's default bounding set drops it — grant
# it explicitly. Note this is NET_ADMIN over the *host* netns; the
# gateway container is trusted infra (it already terminates TLS and
# fronts every vhost), so no new trust boundary is crossed.
additionalCapabilities = [ "CAP_NET_ADMIN" ];
# Bind-mount the per-agent socket dir so nginx inside the gateway
# container can `connect(2)` to the UDS upstreams.
# Read-only (we just connect; harness writes the socket inside
# the agent's own container). Host-side dir is pre-created by a
# tmpfiles rule so nspawn always finds a source at boot.
bindMounts."/run/hive-agent" = {
hostPath = "/run/hive-agent";
isReadOnly = true;
};
# Bind-mount ONLY the gateway-specific subdir of the hyperhive
# state dir. Scoped to /var/lib/hyperhive/gateway/ rather than
# the whole parent so the gateway container can't read forge
# tokens or other files that may live at the parent level.
# c0re writes agents.conf under this subdir and triggers an nginx
# reload from the host via systemd-run after each write.
# Pre-created by a tmpfiles rule.
bindMounts."/run/hive-state" = {
hostPath = "/var/lib/hyperhive/gateway";
isReadOnly = true;
};
# Operator-provided TLS cert dir (e.g. Let's Encrypt / ACME).
# Only mounted when `tls.certDir` is set; when it is, the self-signed
# floor is off (so the `/run/hive-ca` mount below is absent). nginx
# reads cert + key from `/run/hive-tls/<certName>` and `<keyName>`.
bindMounts."/run/hive-tls" = lib.mkIf (cfg.tls.certDir != null) {
hostPath = cfg.tls.certDir;
isReadOnly = true;
};
# Self-signed mode: the host `hive-tls-ca` service generates a hive
# CA + a leaf signed by it under `services.hyperhive.tls.stateDir`.
# Bind-mount that dir read-only so the in-container import service
# (below) can copy the leaf into nginx's state dir with the right
# owner/mode. Source files: `gateway.pem` + `gateway-key.pem`.
bindMounts."/run/hive-ca" = lib.mkIf useSelfSigned {
hostPath = config.services.hyperhive.tls.stateDir;
isReadOnly = true;
};
config =
{ pkgs, ... }:
let
tlsDir = "/var/lib/hive-gateway/tls";
# TLS cert + key paths inside the container.
# - self-signed (default): imported hive-CA-signed leaf in the
# persistent state dir.
# - tls.certDir set: operator-provided cert bind-mounted at /run/hive-tls.
tlsCert =
if cfg.tls.certDir != null then "/run/hive-tls/${cfg.tls.certName}" else "${tlsDir}/cert.pem";
tlsKey =
if cfg.tls.certDir != null then "/run/hive-tls/${cfg.tls.keyName}" else "${tlsDir}/key.pem";
# The swarm-services pair, used only by the vhosts whose names
# this hive's CA cannot sign. Self-signed mode only: with an
# operator cert or ACME the operator owns every name and there
# is no second issuer in the picture.
svcCert = "${tlsDir}/swarm-services.pem";
svcKey = "${tlsDir}/swarm-services-key.pem";
nginxTree = import ./vhosts.nix {
inherit
lib
cfg
forgeCfg
matrixCfg
hyperhiveDomain
dashboardDist
swaggerUiTheme
tlsCert
tlsKey
svcCert
svcKey
swarmServiceDomains
;
errorPages = import ./error-pages.nix { inherit pkgs; };
};
in
{
system.stateVersion = "26.05";
# nginx reload is triggered from the HOST side by hive-c0re
# after each agents.conf write, through hive-priv (c0re is
# unprivileged and cannot act on a system unit).
#
# It stays an explicit trigger rather than a systemd path unit
# watching the file. That used to be impossible — an IN_MOVED_TO
# from the atomic rename did not cross the nspawn mount-namespace
# boundary — and with one machine it would now work. It is still
# not wanted: the write and the reload belong in one causal chain
# c0re can retry and report on (see RELOAD_PENDING), not two
# independent units racing on an inotify event.
# This container shares the host netns, so its own
# firewall.service would run against the HOST ruleset: flush
# the nixos-fw chains, rebuild them from this container's
# (empty) port list, and delete the host's nixos-nat-* chains
# — wiping the bridge DHCP/DNS holes and the agents' NAT on
# every container boot. The host firewall owns all filtering;
# never run one in here.
networking.firewall.enable = false;
services.nginx = {
enable = true;
recommendedProxySettings = true;
recommendedTlsSettings = true;
recommendedGzipSettings = true;
recommendedOptimisation = true;
inherit (nginxTree) appendHttpConfig virtualHosts;
};
# Keep the host-copied /etc/resolv.conf intact. nixos-container
# copies the host's file in at every container start, but
# resolvconf's host-tracking mode then regenerates it — to an
# empty file, since the host file doesn't cross the boundary
# after start (the same failure the matrix container hit).
# With resolvconf off, nothing in here touches the copy: nginx's
# own lookups (ACME) and dnsmasq's follow-the-host upstream
# default (see ./dnsmasq.nix) both read the host's resolvers.
# Keeping it stale-free is the host's job — see the
# `hive-gateway-resolv` path unit above.
networking.resolvconf.enable = false;
# ACME (Let's Encrypt) integration. nginx vhosts set
# `enableACME = true` via the vhost builder; this provides the
# shared ACME config (acceptTerms + email). The gateway
# container has shared host netns so outbound ACME requests
# work without extra routing config. Certs are stored in the
# container's persistent state (`ephemeral = false`).
security.acme = lib.mkIf cfg.tls.acme.enable {
acceptTerms = true;
defaults.email = cfg.tls.acme.email;
};
# Import the host-generated leaf cert before nginx starts.
# The hive CA + gateway leaf are generated on the HOST by
# `hive-tls-ca` (see `hive-tls.nix`) and bind-mounted read-only
# at `/run/hive-ca`; this service copies the leaf into nginx's
# state dir with the owner/mode nginx needs, rather than reading
# the bind-mount directly (the host key is 0600 root:root and a
# cross-namespace bind-mount can't be relaxed in place). nginx
# `Requires=` this via `requiredBy`, so it refuses to start until
# the copy succeeds. ALWAYS runs (no ConditionPathExists) and is
# idempotent — necessary to reconcile broken state from prior
# failed boots (a 0700 dir from a stale UMask, a truncated copy
# from an interrupted oneshot, etc.). The leaf covers the bare
# hive domain plus `forge.`, `matrix.` and `*.${hyperhiveDomain}`
# so all sub-domains validate under the same cert + the hive CA.
# See `docs/gateway.md` ("Self-signed TLS").
systemd.services.hive-gateway-self-signed-cert = lib.mkIf useSelfSigned {
description = "Import host-generated TLS leaf for hive-gateway";
wantedBy = [ "multi-user.target" ];
before = [ "nginx.service" ];
requiredBy = [ "nginx.service" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
# Pin the journal identity (else it's the `script` store-path wrapper).
SyslogIdentifier = "hive-gateway-self-signed-cert";
};
path = [ pkgs.coreutils ];
script = ''
set -eu
mkdir -p ${tlsDir}
# 0755 on BOTH the cert dir and its parent so the nginx
# user can traverse the full path. The parent
# `/var/lib/hive-gateway` lands at 0700 by default (systemd
# StateDirectory / mkdir umask depending on which service
# created it first), which on its own blocks traversal.
# Re-applied every boot in case a prior run left a tighter
# mode behind.
chmod 0755 ${builtins.dirOf tlsDir}
chmod 0755 ${tlsDir}
# Copy the host leaf in. `install` writes atomically with the
# target mode; run as root (container root == host root,
# privateUsers=false) so the 0600 root:root host key is
# readable. Key ends up root:nginx 0640 so nginx-pre-start
# (which runs `nginx -t` as the nginx user, not root) can
# read it — a 0600 root:root key passes the master load but
# fails the pre-start config test with `BIO_new_file() …
# Permission denied`, blocking the unit. Cert is world-read.
install -m 0644 /run/hive-ca/gateway.pem ${tlsCert}
install -m 0640 -g nginx /run/hive-ca/gateway-key.pem ${tlsKey}
# The swarm-services leaf, when this host issues one. It is
# a separate pair rather than more SANs on the one above
# because no hive CA can sign these names — each is
# constrained to its own hive's domain and the service
# names are siblings of it.
#
# Absent is a normal state, not a failure: the leaf exists
# only where the swarm CA is autoconfigured, and issuance
# can also fail on a host that wants one.
#
# ⚠️ When it is absent the HIVE leaf goes to this path
# anyway, and that fallback is load-bearing rather than
# tidy. nginx refuses to load a config naming a cert file
# that does not exist — `cannot load certificate … no such
# file` fails the pre-start test, so the vhost does not
# degrade, the ENTIRE gateway dies and takes the forge, the
# dashboard and matrix with it. Serving the hive leaf on a
# swarm-service name is a name mismatch: browsers warn,
# strict clients refuse, everything else keeps working, and
# the operator gets a bad cert instead of no hive.
#
# Measured, not theorised: this exact path took pr1ma's
# gateway down when the services sub-CA failed to issue.
if [ -s /run/hive-ca/swarm-services.pem ]; then
install -m 0644 /run/hive-ca/swarm-services.pem ${svcCert}
install -m 0640 -g nginx /run/hive-ca/swarm-services-key.pem ${svcKey}
else
echo "no swarm-services leaf — serving the hive leaf on those names (mismatch, not an outage)" >&2
install -m 0644 /run/hive-ca/gateway.pem ${svcCert}
install -m 0640 -g nginx /run/hive-ca/gateway-key.pem ${svcKey}
fi
'';
};
# nginx reload is triggered from the HOST side by hive-c0re
# via `systemctl -M hive-gateway reload nginx` after each
# agents.conf write — letting systemd resolve the nginx binary
# path avoids exit-203 EXEC failures. A path unit watching the
# bind-mounted file inside the container does not work: an
# IN_MOVED_TO from an atomic rename on the host does not
# propagate across the nspawn mount-namespace boundary. The
# host-side trigger is the correct approach.
services.nginx = {
enable = true;
recommendedProxySettings = true;
recommendedTlsSettings = true;
recommendedGzipSettings = true;
recommendedOptimisation = true;
inherit (nginxTree) appendHttpConfig virtualHosts;
};
services.dnsmasq = import ./dnsmasq.nix {
inherit
lib
networkCfg
forgeCfg
matrixCfg
hyperhiveDomain
;
};
};
# dnsmasq moves with nginx rather than staying behind: it was only in
# the container because nginx was, and leaving it there would keep the
# whole resolv.conf sync machine alive for a resolver that no longer
# needs it. Host-side it reads the one /etc/resolv.conf directly.
services.dnsmasq = import ./dnsmasq.nix {
inherit
lib
networkCfg
forgeCfg
matrixCfg
hyperhiveDomain
;
};
networking.firewall = lib.mkIf cfg.openFirewall {

10
nix/host-modules/hive-gateway/options.nix
View file

@ -134,9 +134,9 @@ in
and uses this cert, overriding the self-signed default — the
auto-generated hive-CA-signed leaf is skipped entirely.
The directory is bind-mounted read-only into the gateway
container at `/run/hive-tls/`. nginx reads
`<certDir>/<tls.certName>` and `<certDir>/<tls.keyName>`.
nginx reads `<certDir>/<tls.certName>` and
`<certDir>/<tls.keyName>` directly — it runs on the host, so
the directory needs no bind mount and no copy.
Default filenames (`cert.pem` / `key.pem`) match the output
layout of nixpkgs's `security.acme` module.
@ -232,9 +232,7 @@ in
enabled, every request to the gateway's main vhost requires a
valid username and password. nginx's built-in `auth_basic`
module validates credentials against
`/var/lib/hyperhive/gateway/gateway.htpasswd` on the host
(exposed as `/run/hive-state/gateway.htpasswd` inside the
container via the existing gateway state bind-mount). Off by default.
`/var/lib/hyperhive/gateway/gateway.htpasswd`. Off by default.
Manage users with `hivectl gateway create-user`, `delete-user`,
and `list-users` — see `hivectl gateway --help` for usage.

11
nix/host-modules/hive-gateway/vhosts.nix
View file

@ -241,7 +241,7 @@ let
# `/agent/` catch-all 404 + the two internal error-page targets it
# points at. Per-agent `location /agent/<name>/` blocks live in the
# runtime-generated `/run/hive-state/agents.conf` (included via
# runtime-generated `/var/lib/hyperhive/gateway/agents.conf` (included via
# `extraConfig` on the vhost); nginx longest-prefix-match makes a
# real `/agent/<name>/` beat this catch-all. `internal` keeps the
# error pages reachable only through nginx's error handling.
@ -275,7 +275,7 @@ let
# secret (`X-Hub-Signature-256`) protects those endpoints instead.
dashboardAuth = lib.optionalString cfg.auth.enable ''
auth_basic "${cfg.auth.realm}";
auth_basic_user_file /run/hive-state/gateway.htpasswd;
auth_basic_user_file /var/lib/hyperhive/gateway/gateway.htpasswd;
# `=401` keeps the status 401 so the login dialog shows; the
# internal page explains `hivectl gateway create-user`.
error_page 401 =401 /__hive_auth_unauthorized;
@ -397,15 +397,14 @@ in
};
# Per-agent location blocks, generated at runtime by
# hive-c0re and written to /var/lib/hyperhive/gateway/agents.conf
# on the host. The bind-mount at /run/hive-state/ exposes
# that file here. nginx parses `include` at config-load
# time so a reload (triggered by c0re via systemd-run
# on the host — the same machine nginx runs on. nginx parses
# `include` at config-load time so a reload (triggered by c0re
# after each agents.conf write) picks up new or removed
# agents without a nixos-rebuild. nginx's longest-prefix-
# match rule ensures `/agent/<name>/` from this file beats
# the `/agent/` catch-all above.
extraConfig = securityHeaders + ''
include /run/hive-state/agents.conf;
include /var/lib/hyperhive/gateway/agents.conf;
'';
};
}