docs(gateway): describe what is, not what changed
Per review: docs represent current state. Every "used to" / "no longer" clause this branch introduced is gone — including the History section in network.md, which was a whole subsection about a sync mechanism that doesn't exist. Where the removed clause was carrying a real constraint, the constraint stays and is stated in the present tense instead of as a delta: nothing narrows what the gateway's nginx can reach except the directory permissions in front of a socket, and nothing bounds `ReloadGatewayNginx` except the hard-coded unit name. Those read as rules now rather than as the story of how they came to be rules.
This commit is contained in:
parent
07852cabc1
commit
98d895cf9e
15 changed files with 77 additions and 146 deletions
|
|
@ -121,25 +121,18 @@ The mode is load-bearing, not cosmetic. Write permission on a
|
|||
them, and the sticky bit is the only thing that would restrain that (it
|
||||
is not set here). A world-writable socket dir therefore lets anything
|
||||
able to reach the path delete an agent's socket and bind its own — and
|
||||
nginx reaches all of `/run/hive-agent` (as a plain host path since the
|
||||
gateway moved out of its container; it used to be bind-mounted in, which
|
||||
was the same reach through a longer route). Dropping `o=w` removes that
|
||||
permission rather than qualifying it.
|
||||
nginx reaches all of `/run/hive-agent` as a plain host path. Dropping
|
||||
`o=w` removes that permission rather than qualifying it.
|
||||
|
||||
⚠️ **The gateway leaving its container is a deliberate trade, recorded
|
||||
here so it is not mistaken for an oversight.** nginx and dnsmasq run on
|
||||
the host next to `hive-c0re` (see `docs/gateway.md`). What was given up
|
||||
is a *mount/pid* namespace — **not** a network one: that container ran
|
||||
with `privateNetwork = false` and shared the host's netns, so nginx was
|
||||
already binding host ports and already reaching `localhost` upstreams.
|
||||
The boundary bought no network isolation while costing a resolv.conf
|
||||
sync, a reload that had to cross the machine bus, and three bind mounts.
|
||||
🔑 It did cost one real thing, and the replacement is explicit: the
|
||||
privileged reload verb used to be scoped by `--machine=hive-gateway`,
|
||||
which could only ever reach into that one container. With no namespace
|
||||
to bound it, the unit name is hard-coded in `hive-priv` instead — see
|
||||
`PrivRequest::ReloadGatewayNginx`. **A caller cannot name the unit, so
|
||||
the verb cannot be steered at another service.**
|
||||
⚠️ **The gateway's nginx and dnsmasq are host services, next to
|
||||
`hive-c0re`** (see `docs/gateway.md`) — there is no namespace between
|
||||
them and the rest of the host. That costs no network isolation: nginx
|
||||
binds the host's `:80`/`:443` and reaches `localhost` upstreams, which a
|
||||
netns would have to be opened up for anyway.
|
||||
🔑 It does mean nothing *implicitly* scopes the privileged reload verb,
|
||||
so the scope is explicit: the unit name is hard-coded in `hive-priv` —
|
||||
see `PrivRequest::ReloadGatewayNginx`. **A caller cannot name the unit,
|
||||
so the verb cannot be steered at another service.**
|
||||
|
||||
⚠️ Contrast `/shared`, which *is* sticky world-writable (`1777`): it has
|
||||
many legitimate writers, so sticky is the best available answer there.
|
||||
|
|
|
|||
|
|
@ -122,12 +122,9 @@ now set unconditionally for every agent. The mechanism:
|
|||
- failed → `systemctl reset-failed nginx` + `systemctl start nginx`
|
||||
- otherwise → `systemctl start nginx`
|
||||
This is an explicit trigger rather than a path unit watching the
|
||||
file. It used to be *impossible* to do it any other way — `IN_MOVED_TO`
|
||||
from the atomic rename did not cross the nspawn mount-namespace
|
||||
boundary, so an in-container path unit never fired. With nginx on the
|
||||
host a path unit would now work, and it is still not wanted: the write
|
||||
and the reload belong in one causal chain c0re can retry and report on
|
||||
(`RELOAD_PENDING`), not two units racing on an inotify event.
|
||||
file: the write and the reload belong in one causal chain c0re can
|
||||
retry and report on (`RELOAD_PENDING`), not two units racing on an
|
||||
inotify event.
|
||||
|
||||
c0re regenerates `agents.conf` (and triggers a reload) on two
|
||||
triggers: every topology change (new/removed agents) and every 10s
|
||||
|
|
@ -200,7 +197,7 @@ On by default, and listens on `httpsPort` (default 443) on every vhost beside th
|
|||
|
||||
The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host service (`hive-tls-ca.service`, from the `hive-tls` module) generates a long-lived CA (`services.hyperhive.tls.caValidityDays`, default ~20y) under `services.hyperhive.tls.stateDir` (default `/var/lib/hive-tls`), then signs a gateway **leaf** (`leafValidityDays`, default 30d) with it. `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`).
|
||||
|
||||
⚠️ **That import unit is not a leftover of the old container — do not collapse it into pointing nginx at the CA dir.** It does two jobs. It re-modes the leaf (`hive-tls-ca` writes the key `0600 root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a `0600` key fails the config test and blocks the unit), and it guarantees **every cert path the nginx config names exists** — which is what the swarm-services fallback below is for. Removing it re-creates the #3097 outage.
|
||||
⚠️ **Do not collapse that import unit into pointing nginx at the CA dir.** It does two jobs. It re-modes the leaf (`hive-tls-ca` writes the key `0600 root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a `0600` key fails the config test and blocks the unit), and it guarantees **every cert path the nginx config names exists** — which is what the swarm-services fallback below is for. Removing it re-creates the #3097 outage.
|
||||
|
||||
**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them.
|
||||
|
||||
|
|
@ -433,9 +430,7 @@ any content change — so `max-age=31536000` is safe.
|
|||
**Why the nix store path resolves**: `HIVE_AGENT_FRONTEND_DIR` is a nix
|
||||
store path baked in at hive-c0re build time, and c0re (writing
|
||||
`agents.conf`) and nginx (serving files from it) are on the same machine,
|
||||
so they see the same store. This used to need explaining — nspawn
|
||||
bind-mounts `/nix/store` read-only into a container, which is what made
|
||||
the baked-in path work from inside the gateway.
|
||||
so they see the same store.
|
||||
|
||||
**Graceful degradation**: if `HIVE_AGENT_FRONTEND_DIR` is empty or
|
||||
unset (e.g. a build that predates `cfg.frontend`), each agent gets the
|
||||
|
|
|
|||
|
|
@ -124,37 +124,10 @@ There is deliberately no fallback `server=`: dnsmasq queries all known
|
|||
upstreams in parallel, so a hardcoded public resolver would take a share
|
||||
of normal traffic, not just cover the gap.
|
||||
|
||||
### History: the resolv.conf sync, and why it is gone
|
||||
|
||||
Until the gateway moved onto the host, dnsmasq ran in the `hive-gateway`
|
||||
container and read *that* container's `/etc/resolv.conf` — a one-shot
|
||||
copy nixos-container made at start. systemd-nspawn(1) is explicit that
|
||||
nothing propagates into it after early init, because resolv.conf is
|
||||
normally updated by rename rather than in place. So a host network change
|
||||
(new router, new lease, laptop moving networks) stranded dnsmasq on a
|
||||
resolver that no longer answered, and every non-hive lookup from every
|
||||
agent hung until someone restarted the gateway.
|
||||
|
||||
A host-side `hive-gateway-resolv` path unit closed that gap: watch
|
||||
`/etc/resolv.conf`, `machinectl copy-to` it into the container, reload
|
||||
dnsmasq. Roughly eighty lines of watcher, marker file, is-active guard
|
||||
and mid-rewrite-snapshot check — **all of it bridging two copies of one
|
||||
file.** With one machine there is one file, and the whole unit is
|
||||
deleted.
|
||||
|
||||
🔑 Worth keeping as a shape, not just a story: **the sync was not
|
||||
complexity anyone chose. It was the cost of a boundary that bought
|
||||
nothing here** — the gateway already ran with `privateNetwork = false`,
|
||||
sharing the host's netns, so the container never provided network
|
||||
isolation in the first place. When a workaround is that elaborate, the
|
||||
question to ask is what the boundary is *for*.
|
||||
|
||||
(Two alternatives were considered at the time and both were worse than
|
||||
the copy: a path unit *inside* the container never fired, because the
|
||||
host replaces the file by rename and `IN_MOVED_TO` does not cross the
|
||||
nspawn mount namespace; and bind-mounting the host's `/etc/resolv.conf`
|
||||
would have pinned the *first* inode for the container's whole lifetime,
|
||||
since openresolv writes a temp file and renames over the target.)
|
||||
dnsmasq runs on the host and reads the host's `/etc/resolv.conf`
|
||||
directly, so a network change (new router, new lease, laptop moving
|
||||
networks) reaches it the moment openresolv rewrites the file. There is
|
||||
nothing to synchronise and no unit watching for it.
|
||||
|
||||
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
|
||||
listener only accepts queries from the bridge interface (plus lo for
|
||||
|
|
|
|||
|
|
@ -191,16 +191,12 @@ directly invoke `nixos-container`, `journalctl -M`, or act on a system
|
|||
unit (`systemctl reload nginx`) — those require root. `hive-priv` fills
|
||||
this gap.
|
||||
|
||||
⚠️ **Note what that costs when a helper verb loses its namespace.**
|
||||
`ReloadGatewayNginx` used to run `systemctl -M hive-gateway …`, and the
|
||||
`--machine=` flag was doing two jobs: it was the *transport* into the
|
||||
container **and** the *scope* — the verb could not reach anything
|
||||
outside that one machine. With nginx on the host the transport is
|
||||
unnecessary and the scope went with it, so the containment is now the
|
||||
hard-coded unit name in `hive-priv`: a caller cannot name the unit, so
|
||||
the verb cannot be steered at another service. **When a privileged verb
|
||||
stops needing a namespace, check whether the namespace was also what
|
||||
bounded it.**
|
||||
⚠️ **`ReloadGatewayNginx` acts on a host unit, so nothing implicitly
|
||||
scopes it.** Its containment is the unit name hard-coded in `hive-priv`:
|
||||
a caller cannot name the unit, so the verb cannot be steered at another
|
||||
service. **A privileged verb needs something bounding what it can act
|
||||
on; when that isn't a namespace, it has to be a constant the caller
|
||||
can't supply.**
|
||||
|
||||
### hive-priv
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue