docs(gateway): describe what is, not what changed

Per review: docs represent current state. Every "used to" / "no longer"
clause this branch introduced is gone — including the History section in
network.md, which was a whole subsection about a sync mechanism that
doesn't exist.

Where the removed clause was carrying a real constraint, the constraint
stays and is stated in the present tense instead of as a delta: nothing
narrows what the gateway's nginx can reach except the directory
permissions in front of a socket, and nothing bounds `ReloadGatewayNginx`
except the hard-coded unit name. Those read as rules now rather than as
the story of how they came to be rules.
This commit is contained in:
atlas 2026-08-11 18:09:51 +02:00
commit 98d895cf9e
15 changed files with 77 additions and 146 deletions

View file

@ -121,25 +121,18 @@ The mode is load-bearing, not cosmetic. Write permission on a
them, and the sticky bit is the only thing that would restrain that (it
is not set here). A world-writable socket dir therefore lets anything
able to reach the path delete an agent's socket and bind its own — and
nginx reaches all of `/run/hive-agent` (as a plain host path since the
gateway moved out of its container; it used to be bind-mounted in, which
was the same reach through a longer route). Dropping `o=w` removes that
permission rather than qualifying it.
nginx reaches all of `/run/hive-agent` as a plain host path. Dropping
`o=w` removes that permission rather than qualifying it.
⚠️ **The gateway leaving its container is a deliberate trade, recorded
here so it is not mistaken for an oversight.** nginx and dnsmasq run on
the host next to `hive-c0re` (see `docs/gateway.md`). What was given up
is a *mount/pid* namespace — **not** a network one: that container ran
with `privateNetwork = false` and shared the host's netns, so nginx was
already binding host ports and already reaching `localhost` upstreams.
The boundary bought no network isolation while costing a resolv.conf
sync, a reload that had to cross the machine bus, and three bind mounts.
🔑 It did cost one real thing, and the replacement is explicit: the
privileged reload verb used to be scoped by `--machine=hive-gateway`,
which could only ever reach into that one container. With no namespace
to bound it, the unit name is hard-coded in `hive-priv` instead — see
`PrivRequest::ReloadGatewayNginx`. **A caller cannot name the unit, so
the verb cannot be steered at another service.**
⚠️ **The gateway's nginx and dnsmasq are host services, next to
`hive-c0re`** (see `docs/gateway.md`) — there is no namespace between
them and the rest of the host. That costs no network isolation: nginx
binds the host's `:80`/`:443` and reaches `localhost` upstreams, which a
netns would have to be opened up for anyway.
🔑 It does mean nothing *implicitly* scopes the privileged reload verb,
so the scope is explicit: the unit name is hard-coded in `hive-priv`
see `PrivRequest::ReloadGatewayNginx`. **A caller cannot name the unit,
so the verb cannot be steered at another service.**
⚠️ Contrast `/shared`, which *is* sticky world-writable (`1777`): it has
many legitimate writers, so sticky is the best available answer there.

View file

@ -122,12 +122,9 @@ now set unconditionally for every agent. The mechanism:
- failed → `systemctl reset-failed nginx` + `systemctl start nginx`
- otherwise → `systemctl start nginx`
This is an explicit trigger rather than a path unit watching the
file. It used to be *impossible* to do it any other way — `IN_MOVED_TO`
from the atomic rename did not cross the nspawn mount-namespace
boundary, so an in-container path unit never fired. With nginx on the
host a path unit would now work, and it is still not wanted: the write
and the reload belong in one causal chain c0re can retry and report on
(`RELOAD_PENDING`), not two units racing on an inotify event.
file: the write and the reload belong in one causal chain c0re can
retry and report on (`RELOAD_PENDING`), not two units racing on an
inotify event.
c0re regenerates `agents.conf` (and triggers a reload) on two
triggers: every topology change (new/removed agents) and every 10s
@ -200,7 +197,7 @@ On by default, and listens on `httpsPort` (default 443) on every vhost beside th
The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host service (`hive-tls-ca.service`, from the `hive-tls` module) generates a long-lived CA (`services.hyperhive.tls.caValidityDays`, default ~20y) under `services.hyperhive.tls.stateDir` (default `/var/lib/hive-tls`), then signs a gateway **leaf** (`leafValidityDays`, default 30d) with it. `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`).
⚠️ **That import unit is not a leftover of the old container — do not collapse it into pointing nginx at the CA dir.** It does two jobs. It re-modes the leaf (`hive-tls-ca` writes the key `0600 root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a `0600` key fails the config test and blocks the unit), and it guarantees **every cert path the nginx config names exists** — which is what the swarm-services fallback below is for. Removing it re-creates the #3097 outage.
⚠️ **Do not collapse that import unit into pointing nginx at the CA dir.** It does two jobs. It re-modes the leaf (`hive-tls-ca` writes the key `0600 root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a `0600` key fails the config test and blocks the unit), and it guarantees **every cert path the nginx config names exists** — which is what the swarm-services fallback below is for. Removing it re-creates the #3097 outage.
**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them.
@ -433,9 +430,7 @@ any content change — so `max-age=31536000` is safe.
**Why the nix store path resolves**: `HIVE_AGENT_FRONTEND_DIR` is a nix
store path baked in at hive-c0re build time, and c0re (writing
`agents.conf`) and nginx (serving files from it) are on the same machine,
so they see the same store. This used to need explaining — nspawn
bind-mounts `/nix/store` read-only into a container, which is what made
the baked-in path work from inside the gateway.
so they see the same store.
**Graceful degradation**: if `HIVE_AGENT_FRONTEND_DIR` is empty or
unset (e.g. a build that predates `cfg.frontend`), each agent gets the

View file

@ -124,37 +124,10 @@ There is deliberately no fallback `server=`: dnsmasq queries all known
upstreams in parallel, so a hardcoded public resolver would take a share
of normal traffic, not just cover the gap.
### History: the resolv.conf sync, and why it is gone
Until the gateway moved onto the host, dnsmasq ran in the `hive-gateway`
container and read *that* container's `/etc/resolv.conf` — a one-shot
copy nixos-container made at start. systemd-nspawn(1) is explicit that
nothing propagates into it after early init, because resolv.conf is
normally updated by rename rather than in place. So a host network change
(new router, new lease, laptop moving networks) stranded dnsmasq on a
resolver that no longer answered, and every non-hive lookup from every
agent hung until someone restarted the gateway.
A host-side `hive-gateway-resolv` path unit closed that gap: watch
`/etc/resolv.conf`, `machinectl copy-to` it into the container, reload
dnsmasq. Roughly eighty lines of watcher, marker file, is-active guard
and mid-rewrite-snapshot check — **all of it bridging two copies of one
file.** With one machine there is one file, and the whole unit is
deleted.
🔑 Worth keeping as a shape, not just a story: **the sync was not
complexity anyone chose. It was the cost of a boundary that bought
nothing here** — the gateway already ran with `privateNetwork = false`,
sharing the host's netns, so the container never provided network
isolation in the first place. When a workaround is that elaborate, the
question to ask is what the boundary is *for*.
(Two alternatives were considered at the time and both were worse than
the copy: a path unit *inside* the container never fired, because the
host replaces the file by rename and `IN_MOVED_TO` does not cross the
nspawn mount namespace; and bind-mounting the host's `/etc/resolv.conf`
would have pinned the *first* inode for the container's whole lifetime,
since openresolv writes a temp file and renames over the target.)
dnsmasq runs on the host and reads the host's `/etc/resolv.conf`
directly, so a network change (new router, new lease, laptop moving
networks) reaches it the moment openresolv rewrites the file. There is
nothing to synchronise and no unit watching for it.
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
listener only accepts queries from the bridge interface (plus lo for

View file

@ -191,16 +191,12 @@ directly invoke `nixos-container`, `journalctl -M`, or act on a system
unit (`systemctl reload nginx`) — those require root. `hive-priv` fills
this gap.
⚠️ **Note what that costs when a helper verb loses its namespace.**
`ReloadGatewayNginx` used to run `systemctl -M hive-gateway …`, and the
`--machine=` flag was doing two jobs: it was the *transport* into the
container **and** the *scope* — the verb could not reach anything
outside that one machine. With nginx on the host the transport is
unnecessary and the scope went with it, so the containment is now the
hard-coded unit name in `hive-priv`: a caller cannot name the unit, so
the verb cannot be steered at another service. **When a privileged verb
stops needing a namespace, check whether the namespace was also what
bounded it.**
⚠️ **`ReloadGatewayNginx` acts on a host unit, so nothing implicitly
scopes it.** Its containment is the unit name hard-coded in `hive-priv`:
a caller cannot name the unit, so the verb cannot be steered at another
service. **A privileged verb needs something bounding what it can act
on; when that isn't a namespace, it has to be a constant the caller
can't supply.**
### hive-priv

View file

@ -172,9 +172,8 @@ pub async fn serve(
/// `0o666` — world-accessible so the gateway's nginx process can
/// `connect(2)` without sharing a group with the agent user. What bounds
/// that is the per-agent subdir (`/run/hive-agent/<name>/`): the socket
/// mode grants everyone, the directory decides who gets to ask. It used
/// to be bind-mounted into the one container that needed it; nginx is a
/// host unit now, so the directory's own permissions are the whole story.
/// mode grants everyone, so the directory's permissions decide who gets
/// to ask.
///
/// Marker-gating + the gateway-side consumer: see
/// [`docs/gateway.md::Per-agent unix-socket upstream`](../../../docs/gateway.md).

View file

@ -5,9 +5,9 @@
//! is reached from the dashboard — already fully operator-authenticated —
//! so no capability check is needed here, just the same audit trail.
//!
//! The two surfaces no longer cover the same set: the gateway is the
//! operator's to restart and not an agent's, since nginx on the host fronts
//! every hive service. This endpoint keeps all four.
//! The two surfaces cover different sets: this endpoint takes all four,
//! while the agent path refuses the gateway — nginx on the host fronts
//! every hive service, so bouncing it is the operator's call.
use axum::{
extract::{Path as AxumPath, State},

View file

@ -193,8 +193,7 @@ pub(super) async fn get_journal_host(
axum::extract::Query(q): axum::extract::Query<JournalHostQuery>,
) -> Result<Response, ProblemDetails> {
let lines = q.lines.unwrap_or(500).min(5000);
// `nginx.service` is the gateway: its logs used to live in the
// hive-gateway container's journal and are host-side now.
// `nginx.service` is the gateway — its logs are host-side.
let allowed = ["hive-c0re.service", "hive-priv.service", "nginx.service"];
let unit = match q.unit.as_deref().filter(|s| !s.is_empty()) {
Some(u) => {

View file

@ -167,16 +167,12 @@ fn render(names: &[String], frontend_dir: Option<&str>) -> String {
/// After a successful write, triggers the appropriate nginx action via
/// `hive-priv` (hive-c0re runs unprivileged and cannot act on a system
/// unit): reload when nginx is active, reset-failed+start when in a
/// failed state, plain start otherwise. Writer and nginx are now on the
/// same machine, so this is a plain unit action rather than the old
/// `systemd-run --machine=hive-gateway` hop across the container
/// boundary. It stays an explicit trigger rather than a systemd path
/// unit watching the file. A path unit would now *work* — `IN_MOVED_TO`
/// (fired by the atomic rename) failed to propagate across the nspawn
/// mount-namespace boundary, and that boundary is gone — but it is still
/// not wanted: the write already knows it changed something, and a
/// watcher turns one causal edge into a race with the writer's own
/// rename (see `docs/gateway.md`).
/// failed state, plain start otherwise. Writer and nginx are on the same
/// machine, so it is a plain unit action with no machine-bus hop.
/// It stays an explicit trigger rather than a systemd path
/// unit watching the file: the write already knows it changed something,
/// and a watcher would turn one causal edge into a race with the writer's
/// own rename (see `docs/gateway.md`).
///
/// The priv call is best-effort — a failed sync is logged but not fatal.
/// `reload_if_pending` retries on the next `spawn_poll` tick so a

View file

@ -219,11 +219,11 @@ pub fn shared_root() -> PathBuf {
pub const KNOWLEDGE_DIR: &str = "/var/lib/hyperhive/knowledge";
/// `gateway/` — generated nginx include fragments for the gateway vhost.
/// nginx runs on the host and reads this path directly; it used to be
/// bind-mounted into a gateway container at `/run/hive-state/`, exposing
/// this subdir *only* so the rest of `/var/lib/hyperhive/` (forge/matrix
/// tokens, etc.) stayed out of reach. On the host that narrowing is the
/// unit's sandbox, not a mount — nginx is not confined by this path.
/// nginx runs on the host and reads this path directly. ⚠️ Nothing about
/// this path confines it: what keeps nginx away from the rest of
/// `/var/lib/hyperhive/` (forge/matrix tokens, etc.) is the unit's own
/// sandbox, so widening that sandbox widens what a gateway compromise
/// reaches.
// nix: named by the gateway's nginx config (hive-gateway/vhosts.nix) — must match.
#[must_use]
pub fn gateway_dir() -> PathBuf {

View file

@ -14,11 +14,10 @@ use std::path::{Path, PathBuf};
use anyhow::{Context, Result};
/// Host-side parent directory holding per-agent socket subdirs. The
/// gateway's nginx runs on the host and reads this whole tree, so it
/// can `proxy_pass` to any agent — it used to get there through a
/// read-only bind-mount of the same tree. Each agent's container
/// bind-mounts only its own `<name>/` subdir, which is still what stops
/// one agent reaching another's socket. The literal lives in `hive-host-sock` (shared with
/// gateway's nginx runs on the host and reads this whole tree, so it can
/// `proxy_pass` to any agent. Each agent's container bind-mounts only its
/// own `<name>/` subdir — that mount is what stops one agent reaching
/// another's socket. The literal lives in `hive-host-sock` (shared with
/// `hivectl`); re-exported here under the name this module's consumers
/// have always used.
pub use hive_host_sock::AGENT_SOCKET_DIR;

View file

@ -92,10 +92,9 @@ pub enum InfraContainer {
/// What an [`InfraContainer`] resolves to on the host — i.e. the thing a
/// lifecycle verb actually acts on.
///
/// The gateway is why this exists: its nginx + dnsmasq were lifted out of
/// an nspawn container and onto the host, so "restart the gateway" is a
/// plain host unit now. The operator verb is unchanged; only its target
/// moved. Everything else is still a container.
/// The gateway is why this exists: its nginx + dnsmasq are host
/// services, so "restart the gateway" means a plain host unit. Every
/// other variant is a container.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum InfraTarget {
/// An nspawn container, controlled via `container@<name>.service` and
@ -433,17 +432,13 @@ pub enum PrivRequest {
/// - otherwise → `systemctl start nginx`
///
/// Requires root because hive-c0re runs as the unprivileged
/// `hive-core` user and cannot act on a system unit. It used to be
/// root for a *different* reason — `--machine=hive-gateway` entering
/// the container's namespace over the machine bus — and that reason
/// died with the container: nginx is a host unit now. The
/// requirement survived the move; its justification did not.
/// `hive-core` user and cannot act on a system unit.
///
/// ⚠️ The unit name is **not** a parameter and must stay that way.
/// `--machine=` was doing double duty — transport *and* scope — so
/// dropping it removed the containment along with the namespace hop.
/// Hard-coding `nginx` is what replaces it: a caller cannot name the
/// unit, so this verb cannot be steered at any other service.
/// nginx is a host unit, so nothing else narrows what this verb can
/// touch: hard-coding `nginx` is the entire containment. A caller
/// cannot name the unit, so this verb cannot be steered at any other
/// service.
ReloadGatewayNginx,
// --- Forge admin CLI ---

View file

@ -2186,11 +2186,9 @@ async fn read_container_journal(container: &str, query: &JournalQuery) -> Result
/// - otherwise → `systemctl start nginx`
///
/// ⚠️ `nginx` is hard-coded on purpose — see `PrivRequest::ReloadGatewayNginx`.
/// The unit name is the scope of this verb, and it used to be enforced by
/// `--machine=hive-gateway` (which could only reach into that container).
/// With nginx on the host there is no namespace to bound it, so the
/// literal is the only thing standing between "reload the gateway" and
/// "reload anything".
/// The unit name is the scope of this verb: nginx is a host unit, so no
/// namespace bounds it and the literal is the only thing standing between
/// "reload the gateway" and "reload anything".
///
/// Returns `(String::new(), String::new())` on success so it fits the
/// `exec` return type directly.

View file

@ -1,10 +1,9 @@
# Single nginx in front of every hyperhive web surface — dashboard,
# per-agent UIs (sub-path), forge + matrix (sub-domain), .well-known
# delegations — plus the hive-internal dnsmasq resolver. Both run on the
# HOST, next to hive-c0re. They used to live in a `hive-gateway`
# container that shared the host netns anyway, so the boundary bought no
# network isolation and cost a resolv.conf sync, a reload that had to
# cross the machine bus, and four bind mounts.
# HOST, next to hive-c0re: nginx binds the host's :80/:443 and dnsmasq
# answers on the hive bridge, so neither can be confined to a network
# namespace of its own.
# Full vhost map + discovery flow + design rationale in
# `docs/gateway.md`. Layout: ./options.nix (option declarations),
# ./vhosts.nix (the nginx virtual-host tree), ./error-pages.nix
@ -106,11 +105,9 @@ in
];
# Ensure the gateway state dirs exist at host boot, before anything
# reads or writes them. They used to double as bind-mount sources
# for the container (nspawn would auto-create a missing one); the
# rules stay because they still cover the fresh-boot window before
# c0re has run, and they pin owner + mode rather than leaving it to
# whoever creates the path first.
# reads or writes them: these rules cover the fresh-boot window
# before c0re has run, and pin owner + mode rather than leaving it
# to whoever creates the path first.
#
# /run/hive-agent — per-agent UDS socket dir, written by c0re's
# set_nspawn_flags when agents start. Owned by `hive-core` (the
@ -273,11 +270,8 @@ in
# unprivileged and cannot act on a system unit).
#
# It stays an explicit trigger rather than a systemd path unit
# watching the file. That used to be impossible — an IN_MOVED_TO
# from the atomic rename did not cross the nspawn mount-namespace
# boundary — and with one machine it would now work. It is still
# not wanted: the write and the reload belong in one causal chain
# c0re can retry and report on (see RELOAD_PENDING), not two
# watching the file: the write and the reload belong in one causal
# chain c0re can retry and report on (see RELOAD_PENDING), not two
# independent units racing on an inotify event.
services.nginx = {
@ -289,10 +283,9 @@ in
inherit (nginxTree) appendHttpConfig virtualHosts;
};
# dnsmasq moves with nginx rather than staying behind: it was only in
# the container because nginx was, and leaving it there would keep the
# whole resolv.conf sync machine alive for a resolver that no longer
# needs it. Host-side it reads the one /etc/resolv.conf directly.
# dnsmasq is a host service alongside nginx, so it reads the host's
# /etc/resolv.conf directly and picks up network changes as they
# happen — no copy to keep in sync.
services.dnsmasq = import ./dnsmasq.nix {
inherit
lib

View file

@ -42,11 +42,9 @@ What keeps that safe is that the directory holds **one** socket. So:
> Pointing nginx at that directory to reach this socket would put the admin
> socket within its reach too.
This got *less* forgiving when nginx moved onto the host: the gateway used to
reach a unix upstream through a bind-mount, so the mount list was a second
bound on what it could touch. There is no mount now — the directory is the
whole of the access control. A unit test pins the default path so a tidying
edit fails instead of reviewing cleanly.
nginx is a host service, so nothing narrows what it can reach except the
directory itself — that is the whole of the access control. A unit test pins
the default path so a tidying edit fails instead of reviewing cleanly.
`RuntimeDirectoryPreserve=yes` and the daemon's stale-socket unlink on start are
a **pair**: preserving the directory without the unlink means `bind` fails with

View file

@ -29,10 +29,8 @@ use axum::{Router, routing::get};
/// The directory is its own — deliberately not shared with hive-c0re's
/// `/run/hyperhive`. The socket is `0666`, so its directory is the only
/// access control it has; co-locating it with c0re's admin socket would
/// put both within reach of whatever can reach either. That used to be
/// enforced by which *directory* was bind-mounted into the gateway
/// container; with nginx on the host the mount is gone and the directory
/// is all that is left, so the rule matters more, not less.
/// put both within reach of whatever can reach either. nginx runs on the
/// host, so nothing narrows its reach for you.
const DEFAULT_SOCKET: &str = "/run/swarm-controller/controller.sock";
fn socket_path() -> PathBuf {
@ -98,9 +96,8 @@ mod tests {
/// The socket must not share a directory with anything else, because
/// the socket is `0666` and the directory is therefore the only access
/// control it has. `/run/hyperhive` in particular holds hive-c0re's
/// **admin** socket. nginx used to reach a unix upstream by mounting
/// the socket's whole directory into the gateway container; it runs on
/// the host now, so nothing narrows its reach but the directory itself.
/// **admin** socket, and nginx — a host service — is bounded by
/// nothing but the directory itself.
///
/// A test rather than a comment: the failure this guards against is a
/// one-word edit that looks tidier and reads fine in review.