feat(3088): move the gateway's nginx + dnsmasq onto the host
The gateway's nginx + dnsmasq no longer run in their own nspawn container. `nix/host-modules/hive-gateway/default.nix` loses the `containers.hive-gateway` wrapper and everything that existed only to punch holes in it: `privateNetwork = false`, `CAP_NET_ADMIN`, five bind mounts, its own `stateVersion`, `networking.firewall.enable = false`, `networking.resolvconf.enable = false`, and the `hive-gateway-resolv` path+service pair. 465 -> 303 lines. The container never bought isolation here. It shared the host netns by necessity — nginx binds the host's :80/:443, dnsmasq answers on the bridge — so each of those settings was undoing a boundary the gateway could not afford in the first place. Four things made it more than a deletion, none of them visible in the nix diff: - The self-signed cert service also imports the hive CA leaf, so removing it with the container would have left nginx naming a missing cert file, which it refuses to load at all. - The nginx reload is a hive-priv verb. It still needs root, but no longer for the reason its doc gave, and `--machine=` was both transport and scope — so the unit name is now hard-coded in the helper as the containment. - The lifecycle verb named a container that stops existing. - `journalctl -M hive-gateway` had no machine to enter. Per the operator's ruling, the operator verb keeps working and agents lose it. `InfraContainer` answered three questions that used to share an answer; it now splits into `name()` (identity), `target()` (Container vs HostUnit), `service_unit()` (the systemd unit), and `agent_restartable()`, which the MCP restart path checks before the capability so the refusal cannot read as "ask for infra_admin". `SIBLING_CONTAINERS` drops the gateway — it gates the requests that name a container as a string — while `FromStr` still accepts it, because that answers what a name is, not who may act on it. The dashboard's gateway journal reads host journald filtered to `nginx.service`. Prose was corrected where it only named a location, and re-argued where the container was doing security work: a `0666` per-agent socket was safe because only the gateway container had the directory bind-mounted. There is no mount now, so the directory permissions are the whole of the access control — the constraint holds, its mechanism doesn't. Gate: nix fmt / clippy --all-targets -D warnings / cargo test all clean (710 tests); hivectl-cli.md regenerated from the clap tree. The nix eval was run in both TLS shapes at this commit: every delta in the rendered virtualHosts is one of the three intended path moves, dnsmasq settings are byte-identical, and the absence probe flips true -> false with bindMounts emptied.
This commit is contained in:
parent
cae2cf8df6
commit
07852cabc1
34 changed files with 704 additions and 618 deletions
|
|
@ -115,49 +115,46 @@ schemes pick their own.
|
|||
dnsmasq is **authoritative** for the hive's own zones — answers
|
||||
`<hive-domain>`, `forge.<hive-domain>`, `matrix.<hive-domain>`
|
||||
queries with the bridge IP (where nginx is reachable). Everything
|
||||
else is forwarded to the host's own resolvers: dnsmasq reads the
|
||||
gateway container's `/etc/resolv.conf`, the host copy nixos-container
|
||||
makes at each container start. Containers don't need to know the
|
||||
upstream — they query the bridge IP and dnsmasq does the right thing
|
||||
per-name.
|
||||
else is forwarded to the host's own resolvers: dnsmasq runs on the host
|
||||
and reads the host's `/etc/resolv.conf` directly. Containers don't need
|
||||
to know the upstream — they query the bridge IP and dnsmasq does the
|
||||
right thing per-name.
|
||||
|
||||
That copy is one-shot — systemd-nspawn(1) is explicit that nothing
|
||||
propagates into it after early init, because resolv.conf is normally
|
||||
updated by rename rather than in place. Left alone, a host network
|
||||
change (new router, new lease, laptop moving networks) would strand
|
||||
dnsmasq on a resolver that no longer answers, and every non-hive
|
||||
lookup from every agent would hang until someone restarted the
|
||||
gateway. The host-side **`hive-gateway-resolv`** path unit closes
|
||||
that: it watches `/etc/resolv.conf`, `machinectl copy-to`s it into
|
||||
the container, and reloads dnsmasq (`SIGHUP` — re-read upstreams +
|
||||
flush cache, nothing dropped). The watch is armed before
|
||||
`network-pre.target` so the boot's first DHCP write is caught as well,
|
||||
and the sync also runs once per gateway start to pick up a resolver
|
||||
change that happened while the container was down. A host file with no
|
||||
`nameserver` line is
|
||||
skipped rather than pushed, so a mid-rewrite snapshot can't blank the
|
||||
hive's DNS. There is deliberately no fallback `server=`: dnsmasq
|
||||
queries all known upstreams in parallel, so a hardcoded public
|
||||
resolver would take a share of normal traffic, not just cover the gap.
|
||||
There is deliberately no fallback `server=`: dnsmasq queries all known
|
||||
upstreams in parallel, so a hardcoded public resolver would take a share
|
||||
of normal traffic, not just cover the gap.
|
||||
|
||||
Two alternatives that look simpler and aren't:
|
||||
### History: the resolv.conf sync, and why it is gone
|
||||
|
||||
- **A path unit inside the container.** The host replaces
|
||||
`/etc/resolv.conf` by rename, and that `IN_MOVED_TO` doesn't cross
|
||||
the nspawn mount namespace — the same reason hive-c0re reloads nginx
|
||||
from the host side after each `agents.conf` write.
|
||||
- **Bind-mounting the host's `/etc/resolv.conf` into the container.**
|
||||
openresolv writes a temp file and renames over the target, so the
|
||||
bind mount would pin the *first* inode for the container's whole
|
||||
lifetime — strictly worse than the copy, which at least a restart
|
||||
clears. (Reachability is not the problem here: the gateway runs with
|
||||
`privateNetwork = false`, so it shares the host's netns and can reach
|
||||
anything the host can.)
|
||||
Until the gateway moved onto the host, dnsmasq ran in the `hive-gateway`
|
||||
container and read *that* container's `/etc/resolv.conf` — a one-shot
|
||||
copy nixos-container made at start. systemd-nspawn(1) is explicit that
|
||||
nothing propagates into it after early init, because resolv.conf is
|
||||
normally updated by rename rather than in place. So a host network change
|
||||
(new router, new lease, laptop moving networks) stranded dnsmasq on a
|
||||
resolver that no longer answered, and every non-hive lookup from every
|
||||
agent hung until someone restarted the gateway.
|
||||
|
||||
`machinectl copy-to` is used rather than writing to the container's
|
||||
rootfs from the host, so the push goes through the container's own
|
||||
mount namespace and stays correct if `/etc` is ever assembled
|
||||
differently (e.g. `system.etc.overlay`).
|
||||
A host-side `hive-gateway-resolv` path unit closed that gap: watch
|
||||
`/etc/resolv.conf`, `machinectl copy-to` it into the container, reload
|
||||
dnsmasq. Roughly eighty lines of watcher, marker file, is-active guard
|
||||
and mid-rewrite-snapshot check — **all of it bridging two copies of one
|
||||
file.** With one machine there is one file, and the whole unit is
|
||||
deleted.
|
||||
|
||||
🔑 Worth keeping as a shape, not just a story: **the sync was not
|
||||
complexity anyone chose. It was the cost of a boundary that bought
|
||||
nothing here** — the gateway already ran with `privateNetwork = false`,
|
||||
sharing the host's netns, so the container never provided network
|
||||
isolation in the first place. When a workaround is that elaborate, the
|
||||
question to ask is what the boundary is *for*.
|
||||
|
||||
(Two alternatives were considered at the time and both were worse than
|
||||
the copy: a path unit *inside* the container never fired, because the
|
||||
host replaces the file by rename and `IN_MOVED_TO` does not cross the
|
||||
nspawn mount namespace; and bind-mounting the host's `/etc/resolv.conf`
|
||||
would have pinned the *first* inode for the container's whole lifetime,
|
||||
since openresolv writes a temp file and renames over the target.)
|
||||
|
||||
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
|
||||
listener only accepts queries from the bridge interface (plus lo for
|
||||
|
|
@ -289,4 +286,4 @@ loopback — the hive-c0re admin socket is a UDS, not TCP.
|
|||
|
||||
## Cross-references
|
||||
|
||||
- `docs/gateway.md` — vhost map + the gateway container's other duties
|
||||
- `docs/gateway.md` — vhost map + the gateway's other duties
|
||||
|
|
|
|||
Loading…
Reference in a new issue