docs(gateway): describe what is, not what changed

Per review: docs represent current state. Every "used to" / "no longer"
clause this branch introduced is gone — including the History section in
network.md, which was a whole subsection about a sync mechanism that
doesn't exist.

Where the removed clause was carrying a real constraint, the constraint
stays and is stated in the present tense instead of as a delta: nothing
narrows what the gateway's nginx can reach except the directory
permissions in front of a socket, and nothing bounds `ReloadGatewayNginx`
except the hard-coded unit name. Those read as rules now rather than as
the story of how they came to be rules.
This commit is contained in:
atlas 2026-08-11 18:09:51 +02:00
commit 98d895cf9e
15 changed files with 77 additions and 146 deletions

View file

@ -121,25 +121,18 @@ The mode is load-bearing, not cosmetic. Write permission on a
them, and the sticky bit is the only thing that would restrain that (it
is not set here). A world-writable socket dir therefore lets anything
able to reach the path delete an agent's socket and bind its own — and
nginx reaches all of `/run/hive-agent` (as a plain host path since the
gateway moved out of its container; it used to be bind-mounted in, which
was the same reach through a longer route). Dropping `o=w` removes that
permission rather than qualifying it.
nginx reaches all of `/run/hive-agent` as a plain host path. Dropping
`o=w` removes that permission rather than qualifying it.
⚠️ **The gateway leaving its container is a deliberate trade, recorded
here so it is not mistaken for an oversight.** nginx and dnsmasq run on
the host next to `hive-c0re` (see `docs/gateway.md`). What was given up
is a *mount/pid* namespace — **not** a network one: that container ran
with `privateNetwork = false` and shared the host's netns, so nginx was
already binding host ports and already reaching `localhost` upstreams.
The boundary bought no network isolation while costing a resolv.conf
sync, a reload that had to cross the machine bus, and three bind mounts.
🔑 It did cost one real thing, and the replacement is explicit: the
privileged reload verb used to be scoped by `--machine=hive-gateway`,
which could only ever reach into that one container. With no namespace
to bound it, the unit name is hard-coded in `hive-priv` instead — see
`PrivRequest::ReloadGatewayNginx`. **A caller cannot name the unit, so
the verb cannot be steered at another service.**
⚠️ **The gateway's nginx and dnsmasq are host services, next to
`hive-c0re`** (see `docs/gateway.md`) — there is no namespace between
them and the rest of the host. That costs no network isolation: nginx
binds the host's `:80`/`:443` and reaches `localhost` upstreams, which a
netns would have to be opened up for anyway.
🔑 It does mean nothing *implicitly* scopes the privileged reload verb,
so the scope is explicit: the unit name is hard-coded in `hive-priv`
see `PrivRequest::ReloadGatewayNginx`. **A caller cannot name the unit,
so the verb cannot be steered at another service.**
⚠️ Contrast `/shared`, which *is* sticky world-writable (`1777`): it has
many legitimate writers, so sticky is the best available answer there.

View file

@ -122,12 +122,9 @@ now set unconditionally for every agent. The mechanism:
- failed → `systemctl reset-failed nginx` + `systemctl start nginx`
- otherwise → `systemctl start nginx`
This is an explicit trigger rather than a path unit watching the
file. It used to be *impossible* to do it any other way — `IN_MOVED_TO`
from the atomic rename did not cross the nspawn mount-namespace
boundary, so an in-container path unit never fired. With nginx on the
host a path unit would now work, and it is still not wanted: the write
and the reload belong in one causal chain c0re can retry and report on
(`RELOAD_PENDING`), not two units racing on an inotify event.
file: the write and the reload belong in one causal chain c0re can
retry and report on (`RELOAD_PENDING`), not two units racing on an
inotify event.
c0re regenerates `agents.conf` (and triggers a reload) on two
triggers: every topology change (new/removed agents) and every 10s
@ -200,7 +197,7 @@ On by default, and listens on `httpsPort` (default 443) on every vhost beside th
The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host service (`hive-tls-ca.service`, from the `hive-tls` module) generates a long-lived CA (`services.hyperhive.tls.caValidityDays`, default ~20y) under `services.hyperhive.tls.stateDir` (default `/var/lib/hive-tls`), then signs a gateway **leaf** (`leafValidityDays`, default 30d) with it. `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`).
⚠️ **That import unit is not a leftover of the old container — do not collapse it into pointing nginx at the CA dir.** It does two jobs. It re-modes the leaf (`hive-tls-ca` writes the key `0600 root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a `0600` key fails the config test and blocks the unit), and it guarantees **every cert path the nginx config names exists** — which is what the swarm-services fallback below is for. Removing it re-creates the #3097 outage.
⚠️ **Do not collapse that import unit into pointing nginx at the CA dir.** It does two jobs. It re-modes the leaf (`hive-tls-ca` writes the key `0600 root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a `0600` key fails the config test and blocks the unit), and it guarantees **every cert path the nginx config names exists** — which is what the swarm-services fallback below is for. Removing it re-creates the #3097 outage.
**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them.
@ -433,9 +430,7 @@ any content change — so `max-age=31536000` is safe.
**Why the nix store path resolves**: `HIVE_AGENT_FRONTEND_DIR` is a nix
store path baked in at hive-c0re build time, and c0re (writing
`agents.conf`) and nginx (serving files from it) are on the same machine,
so they see the same store. This used to need explaining — nspawn
bind-mounts `/nix/store` read-only into a container, which is what made
the baked-in path work from inside the gateway.
so they see the same store.
**Graceful degradation**: if `HIVE_AGENT_FRONTEND_DIR` is empty or
unset (e.g. a build that predates `cfg.frontend`), each agent gets the

View file

@ -124,37 +124,10 @@ There is deliberately no fallback `server=`: dnsmasq queries all known
upstreams in parallel, so a hardcoded public resolver would take a share
of normal traffic, not just cover the gap.
### History: the resolv.conf sync, and why it is gone
Until the gateway moved onto the host, dnsmasq ran in the `hive-gateway`
container and read *that* container's `/etc/resolv.conf` — a one-shot
copy nixos-container made at start. systemd-nspawn(1) is explicit that
nothing propagates into it after early init, because resolv.conf is
normally updated by rename rather than in place. So a host network change
(new router, new lease, laptop moving networks) stranded dnsmasq on a
resolver that no longer answered, and every non-hive lookup from every
agent hung until someone restarted the gateway.
A host-side `hive-gateway-resolv` path unit closed that gap: watch
`/etc/resolv.conf`, `machinectl copy-to` it into the container, reload
dnsmasq. Roughly eighty lines of watcher, marker file, is-active guard
and mid-rewrite-snapshot check — **all of it bridging two copies of one
file.** With one machine there is one file, and the whole unit is
deleted.
🔑 Worth keeping as a shape, not just a story: **the sync was not
complexity anyone chose. It was the cost of a boundary that bought
nothing here** — the gateway already ran with `privateNetwork = false`,
sharing the host's netns, so the container never provided network
isolation in the first place. When a workaround is that elaborate, the
question to ask is what the boundary is *for*.
(Two alternatives were considered at the time and both were worse than
the copy: a path unit *inside* the container never fired, because the
host replaces the file by rename and `IN_MOVED_TO` does not cross the
nspawn mount namespace; and bind-mounting the host's `/etc/resolv.conf`
would have pinned the *first* inode for the container's whole lifetime,
since openresolv writes a temp file and renames over the target.)
dnsmasq runs on the host and reads the host's `/etc/resolv.conf`
directly, so a network change (new router, new lease, laptop moving
networks) reaches it the moment openresolv rewrites the file. There is
nothing to synchronise and no unit watching for it.
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
listener only accepts queries from the bridge interface (plus lo for

View file

@ -191,16 +191,12 @@ directly invoke `nixos-container`, `journalctl -M`, or act on a system
unit (`systemctl reload nginx`) — those require root. `hive-priv` fills
this gap.
⚠️ **Note what that costs when a helper verb loses its namespace.**
`ReloadGatewayNginx` used to run `systemctl -M hive-gateway …`, and the
`--machine=` flag was doing two jobs: it was the *transport* into the
container **and** the *scope* — the verb could not reach anything
outside that one machine. With nginx on the host the transport is
unnecessary and the scope went with it, so the containment is now the
hard-coded unit name in `hive-priv`: a caller cannot name the unit, so
the verb cannot be steered at another service. **When a privileged verb
stops needing a namespace, check whether the namespace was also what
bounded it.**
⚠️ **`ReloadGatewayNginx` acts on a host unit, so nothing implicitly
scopes it.** Its containment is the unit name hard-coded in `hive-priv`:
a caller cannot name the unit, so the verb cannot be steered at another
service. **A privileged verb needs something bounding what it can act
on; when that isn't a namespace, it has to be a constant the caller
can't supply.**
### hive-priv