diff --git a/docs/boundary.md b/docs/boundary.md index fb065c42..c3ba808c 100644 --- a/docs/boundary.md +++ b/docs/boundary.md @@ -121,25 +121,18 @@ The mode is load-bearing, not cosmetic. Write permission on a them, and the sticky bit is the only thing that would restrain that (it is not set here). A world-writable socket dir therefore lets anything able to reach the path delete an agent's socket and bind its own — and -nginx reaches all of `/run/hive-agent` (as a plain host path since the -gateway moved out of its container; it used to be bind-mounted in, which -was the same reach through a longer route). Dropping `o=w` removes that -permission rather than qualifying it. +nginx reaches all of `/run/hive-agent` as a plain host path. Dropping +`o=w` removes that permission rather than qualifying it. -⚠️ **The gateway leaving its container is a deliberate trade, recorded -here so it is not mistaken for an oversight.** nginx and dnsmasq run on -the host next to `hive-c0re` (see `docs/gateway.md`). What was given up -is a *mount/pid* namespace — **not** a network one: that container ran -with `privateNetwork = false` and shared the host's netns, so nginx was -already binding host ports and already reaching `localhost` upstreams. -The boundary bought no network isolation while costing a resolv.conf -sync, a reload that had to cross the machine bus, and three bind mounts. -🔑 It did cost one real thing, and the replacement is explicit: the -privileged reload verb used to be scoped by `--machine=hive-gateway`, -which could only ever reach into that one container. With no namespace -to bound it, the unit name is hard-coded in `hive-priv` instead — see -`PrivRequest::ReloadGatewayNginx`. **A caller cannot name the unit, so -the verb cannot be steered at another service.** +⚠️ **The gateway's nginx and dnsmasq are host services, next to +`hive-c0re`** (see `docs/gateway.md`) — there is no namespace between +them and the rest of the host. That costs no network isolation: nginx +binds the host's `:80`/`:443` and reaches `localhost` upstreams, which a +netns would have to be opened up for anyway. +🔑 It does mean nothing *implicitly* scopes the privileged reload verb, +so the scope is explicit: the unit name is hard-coded in `hive-priv` — +see `PrivRequest::ReloadGatewayNginx`. **A caller cannot name the unit, +so the verb cannot be steered at another service.** ⚠️ Contrast `/shared`, which *is* sticky world-writable (`1777`): it has many legitimate writers, so sticky is the best available answer there. diff --git a/docs/gateway.md b/docs/gateway.md index 8d5f8a66..6bf2df77 100644 --- a/docs/gateway.md +++ b/docs/gateway.md @@ -122,12 +122,9 @@ now set unconditionally for every agent. The mechanism: - failed → `systemctl reset-failed nginx` + `systemctl start nginx` - otherwise → `systemctl start nginx` This is an explicit trigger rather than a path unit watching the - file. It used to be *impossible* to do it any other way — `IN_MOVED_TO` - from the atomic rename did not cross the nspawn mount-namespace - boundary, so an in-container path unit never fired. With nginx on the - host a path unit would now work, and it is still not wanted: the write - and the reload belong in one causal chain c0re can retry and report on - (`RELOAD_PENDING`), not two units racing on an inotify event. + file: the write and the reload belong in one causal chain c0re can + retry and report on (`RELOAD_PENDING`), not two units racing on an + inotify event. c0re regenerates `agents.conf` (and triggers a reload) on two triggers: every topology change (new/removed agents) and every 10s @@ -200,7 +197,7 @@ On by default, and listens on `httpsPort` (default 443) on every vhost beside th The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host service (`hive-tls-ca.service`, from the `hive-tls` module) generates a long-lived CA (`services.hyperhive.tls.caValidityDays`, default ~20y) under `services.hyperhive.tls.stateDir` (default `/var/lib/hive-tls`), then signs a gateway **leaf** (`leafValidityDays`, default 30d) with it. `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`). -⚠️ **That import unit is not a leftover of the old container — do not collapse it into pointing nginx at the CA dir.** It does two jobs. It re-modes the leaf (`hive-tls-ca` writes the key `0600 root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a `0600` key fails the config test and blocks the unit), and it guarantees **every cert path the nginx config names exists** — which is what the swarm-services fallback below is for. Removing it re-creates the #3097 outage. +⚠️ **Do not collapse that import unit into pointing nginx at the CA dir.** It does two jobs. It re-modes the leaf (`hive-tls-ca` writes the key `0600 root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a `0600` key fails the config test and blocks the unit), and it guarantees **every cert path the nginx config names exists** — which is what the swarm-services fallback below is for. Removing it re-creates the #3097 outage. **Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them. @@ -433,9 +430,7 @@ any content change — so `max-age=31536000` is safe. **Why the nix store path resolves**: `HIVE_AGENT_FRONTEND_DIR` is a nix store path baked in at hive-c0re build time, and c0re (writing `agents.conf`) and nginx (serving files from it) are on the same machine, -so they see the same store. This used to need explaining — nspawn -bind-mounts `/nix/store` read-only into a container, which is what made -the baked-in path work from inside the gateway. +so they see the same store. **Graceful degradation**: if `HIVE_AGENT_FRONTEND_DIR` is empty or unset (e.g. a build that predates `cfg.frontend`), each agent gets the diff --git a/docs/network.md b/docs/network.md index a64e50ac..1d2edbd9 100644 --- a/docs/network.md +++ b/docs/network.md @@ -124,37 +124,10 @@ There is deliberately no fallback `server=`: dnsmasq queries all known upstreams in parallel, so a hardcoded public resolver would take a share of normal traffic, not just cover the gap. -### History: the resolv.conf sync, and why it is gone - -Until the gateway moved onto the host, dnsmasq ran in the `hive-gateway` -container and read *that* container's `/etc/resolv.conf` — a one-shot -copy nixos-container made at start. systemd-nspawn(1) is explicit that -nothing propagates into it after early init, because resolv.conf is -normally updated by rename rather than in place. So a host network change -(new router, new lease, laptop moving networks) stranded dnsmasq on a -resolver that no longer answered, and every non-hive lookup from every -agent hung until someone restarted the gateway. - -A host-side `hive-gateway-resolv` path unit closed that gap: watch -`/etc/resolv.conf`, `machinectl copy-to` it into the container, reload -dnsmasq. Roughly eighty lines of watcher, marker file, is-active guard -and mid-rewrite-snapshot check — **all of it bridging two copies of one -file.** With one machine there is one file, and the whole unit is -deleted. - -🔑 Worth keeping as a shape, not just a story: **the sync was not -complexity anyone chose. It was the cost of a boundary that bought -nothing here** — the gateway already ran with `privateNetwork = false`, -sharing the host's netns, so the container never provided network -isolation in the first place. When a workaround is that elaborate, the -question to ask is what the boundary is *for*. - -(Two alternatives were considered at the time and both were worse than -the copy: a path unit *inside* the container never fired, because the -host replaces the file by rename and `IN_MOVED_TO` does not cross the -nspawn mount namespace; and bind-mounting the host's `/etc/resolv.conf` -would have pinned the *first* inode for the container's whole lifetime, -since openresolv writes a temp file and renames over the target.) +dnsmasq runs on the host and reads the host's `/etc/resolv.conf` +directly, so a network change (new router, new lease, laptop moving +networks) reaches it the moment openresolv rewrites the file. There is +nothing to synchronise and no unit watching for it. `bind-interfaces` + `interface = [ bridgeName "lo" ]` means the listener only accepts queries from the bridge interface (plus lo for diff --git a/docs/security.md b/docs/security.md index bc2f7de4..d2051adc 100644 --- a/docs/security.md +++ b/docs/security.md @@ -191,16 +191,12 @@ directly invoke `nixos-container`, `journalctl -M`, or act on a system unit (`systemctl reload nginx`) — those require root. `hive-priv` fills this gap. -⚠️ **Note what that costs when a helper verb loses its namespace.** -`ReloadGatewayNginx` used to run `systemctl -M hive-gateway …`, and the -`--machine=` flag was doing two jobs: it was the *transport* into the -container **and** the *scope* — the verb could not reach anything -outside that one machine. With nginx on the host the transport is -unnecessary and the scope went with it, so the containment is now the -hard-coded unit name in `hive-priv`: a caller cannot name the unit, so -the verb cannot be steered at another service. **When a privileged verb -stops needing a namespace, check whether the namespace was also what -bounded it.** +⚠️ **`ReloadGatewayNginx` acts on a host unit, so nothing implicitly +scopes it.** Its containment is the unit name hard-coded in `hive-priv`: +a caller cannot name the unit, so the verb cannot be steered at another +service. **A privileged verb needs something bounding what it can act +on; when that isn't a namespace, it has to be a constant the caller +can't supply.** ### hive-priv diff --git a/hive-agent/src/web_ui/mod.rs b/hive-agent/src/web_ui/mod.rs index 05a5badc..86974404 100644 --- a/hive-agent/src/web_ui/mod.rs +++ b/hive-agent/src/web_ui/mod.rs @@ -172,9 +172,8 @@ pub async fn serve( /// `0o666` — world-accessible so the gateway's nginx process can /// `connect(2)` without sharing a group with the agent user. What bounds /// that is the per-agent subdir (`/run/hive-agent//`): the socket -/// mode grants everyone, the directory decides who gets to ask. It used -/// to be bind-mounted into the one container that needed it; nginx is a -/// host unit now, so the directory's own permissions are the whole story. +/// mode grants everyone, so the directory's permissions decide who gets +/// to ask. /// /// Marker-gating + the gateway-side consumer: see /// [`docs/gateway.md::Per-agent unix-socket upstream`](../../../docs/gateway.md). diff --git a/hive-c0re/src/dashboard/infra_containers.rs b/hive-c0re/src/dashboard/infra_containers.rs index 45951b09..572adce8 100644 --- a/hive-c0re/src/dashboard/infra_containers.rs +++ b/hive-c0re/src/dashboard/infra_containers.rs @@ -5,9 +5,9 @@ //! is reached from the dashboard — already fully operator-authenticated — //! so no capability check is needed here, just the same audit trail. //! -//! The two surfaces no longer cover the same set: the gateway is the -//! operator's to restart and not an agent's, since nginx on the host fronts -//! every hive service. This endpoint keeps all four. +//! The two surfaces cover different sets: this endpoint takes all four, +//! while the agent path refuses the gateway — nginx on the host fronts +//! every hive service, so bouncing it is the operator's call. use axum::{ extract::{Path as AxumPath, State}, diff --git a/hive-c0re/src/dashboard/journal.rs b/hive-c0re/src/dashboard/journal.rs index dfdd02b3..3898b5c9 100644 --- a/hive-c0re/src/dashboard/journal.rs +++ b/hive-c0re/src/dashboard/journal.rs @@ -193,8 +193,7 @@ pub(super) async fn get_journal_host( axum::extract::Query(q): axum::extract::Query, ) -> Result { let lines = q.lines.unwrap_or(500).min(5000); - // `nginx.service` is the gateway: its logs used to live in the - // hive-gateway container's journal and are host-side now. + // `nginx.service` is the gateway — its logs are host-side. let allowed = ["hive-c0re.service", "hive-priv.service", "nginx.service"]; let unit = match q.unit.as_deref().filter(|s| !s.is_empty()) { Some(u) => { diff --git a/hive-c0re/src/gateway_nginx.rs b/hive-c0re/src/gateway_nginx.rs index e4cbc694..7a9fda1d 100644 --- a/hive-c0re/src/gateway_nginx.rs +++ b/hive-c0re/src/gateway_nginx.rs @@ -167,16 +167,12 @@ fn render(names: &[String], frontend_dir: Option<&str>) -> String { /// After a successful write, triggers the appropriate nginx action via /// `hive-priv` (hive-c0re runs unprivileged and cannot act on a system /// unit): reload when nginx is active, reset-failed+start when in a -/// failed state, plain start otherwise. Writer and nginx are now on the -/// same machine, so this is a plain unit action rather than the old -/// `systemd-run --machine=hive-gateway` hop across the container -/// boundary. It stays an explicit trigger rather than a systemd path -/// unit watching the file. A path unit would now *work* — `IN_MOVED_TO` -/// (fired by the atomic rename) failed to propagate across the nspawn -/// mount-namespace boundary, and that boundary is gone — but it is still -/// not wanted: the write already knows it changed something, and a -/// watcher turns one causal edge into a race with the writer's own -/// rename (see `docs/gateway.md`). +/// failed state, plain start otherwise. Writer and nginx are on the same +/// machine, so it is a plain unit action with no machine-bus hop. +/// It stays an explicit trigger rather than a systemd path +/// unit watching the file: the write already knows it changed something, +/// and a watcher would turn one causal edge into a race with the writer's +/// own rename (see `docs/gateway.md`). /// /// The priv call is best-effort — a failed sync is logged but not fatal. /// `reload_if_pending` retries on the next `spawn_poll` tick so a diff --git a/hive-c0re/src/paths.rs b/hive-c0re/src/paths.rs index 0d26d019..5e8a89e8 100644 --- a/hive-c0re/src/paths.rs +++ b/hive-c0re/src/paths.rs @@ -219,11 +219,11 @@ pub fn shared_root() -> PathBuf { pub const KNOWLEDGE_DIR: &str = "/var/lib/hyperhive/knowledge"; /// `gateway/` — generated nginx include fragments for the gateway vhost. -/// nginx runs on the host and reads this path directly; it used to be -/// bind-mounted into a gateway container at `/run/hive-state/`, exposing -/// this subdir *only* so the rest of `/var/lib/hyperhive/` (forge/matrix -/// tokens, etc.) stayed out of reach. On the host that narrowing is the -/// unit's sandbox, not a mount — nginx is not confined by this path. +/// nginx runs on the host and reads this path directly. ⚠️ Nothing about +/// this path confines it: what keeps nginx away from the rest of +/// `/var/lib/hyperhive/` (forge/matrix tokens, etc.) is the unit's own +/// sandbox, so widening that sandbox widens what a gateway compromise +/// reaches. // nix: named by the gateway's nginx config (hive-gateway/vhosts.nix) — must match. #[must_use] pub fn gateway_dir() -> PathBuf { diff --git a/hive-c0re/src/workers/agent_sockets.rs b/hive-c0re/src/workers/agent_sockets.rs index b7a07b23..8f2673bb 100644 --- a/hive-c0re/src/workers/agent_sockets.rs +++ b/hive-c0re/src/workers/agent_sockets.rs @@ -14,11 +14,10 @@ use std::path::{Path, PathBuf}; use anyhow::{Context, Result}; /// Host-side parent directory holding per-agent socket subdirs. The -/// gateway's nginx runs on the host and reads this whole tree, so it -/// can `proxy_pass` to any agent — it used to get there through a -/// read-only bind-mount of the same tree. Each agent's container -/// bind-mounts only its own `/` subdir, which is still what stops -/// one agent reaching another's socket. The literal lives in `hive-host-sock` (shared with +/// gateway's nginx runs on the host and reads this whole tree, so it can +/// `proxy_pass` to any agent. Each agent's container bind-mounts only its +/// own `/` subdir — that mount is what stops one agent reaching +/// another's socket. The literal lives in `hive-host-sock` (shared with /// `hivectl`); re-exported here under the name this module's consumers /// have always used. pub use hive_host_sock::AGENT_SOCKET_DIR; diff --git a/hive-priv-sock/src/lib.rs b/hive-priv-sock/src/lib.rs index 3256c074..88399f1f 100644 --- a/hive-priv-sock/src/lib.rs +++ b/hive-priv-sock/src/lib.rs @@ -92,10 +92,9 @@ pub enum InfraContainer { /// What an [`InfraContainer`] resolves to on the host — i.e. the thing a /// lifecycle verb actually acts on. /// -/// The gateway is why this exists: its nginx + dnsmasq were lifted out of -/// an nspawn container and onto the host, so "restart the gateway" is a -/// plain host unit now. The operator verb is unchanged; only its target -/// moved. Everything else is still a container. +/// The gateway is why this exists: its nginx + dnsmasq are host +/// services, so "restart the gateway" means a plain host unit. Every +/// other variant is a container. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum InfraTarget { /// An nspawn container, controlled via `container@.service` and @@ -433,17 +432,13 @@ pub enum PrivRequest { /// - otherwise → `systemctl start nginx` /// /// Requires root because hive-c0re runs as the unprivileged - /// `hive-core` user and cannot act on a system unit. It used to be - /// root for a *different* reason — `--machine=hive-gateway` entering - /// the container's namespace over the machine bus — and that reason - /// died with the container: nginx is a host unit now. The - /// requirement survived the move; its justification did not. + /// `hive-core` user and cannot act on a system unit. /// /// ⚠️ The unit name is **not** a parameter and must stay that way. - /// `--machine=` was doing double duty — transport *and* scope — so - /// dropping it removed the containment along with the namespace hop. - /// Hard-coding `nginx` is what replaces it: a caller cannot name the - /// unit, so this verb cannot be steered at any other service. + /// nginx is a host unit, so nothing else narrows what this verb can + /// touch: hard-coding `nginx` is the entire containment. A caller + /// cannot name the unit, so this verb cannot be steered at any other + /// service. ReloadGatewayNginx, // --- Forge admin CLI --- diff --git a/hive-priv/src/main.rs b/hive-priv/src/main.rs index 8d0f09b4..66d1f803 100644 --- a/hive-priv/src/main.rs +++ b/hive-priv/src/main.rs @@ -2186,11 +2186,9 @@ async fn read_container_journal(container: &str, query: &JournalQuery) -> Result /// - otherwise → `systemctl start nginx` /// /// ⚠️ `nginx` is hard-coded on purpose — see `PrivRequest::ReloadGatewayNginx`. -/// The unit name is the scope of this verb, and it used to be enforced by -/// `--machine=hive-gateway` (which could only reach into that container). -/// With nginx on the host there is no namespace to bound it, so the -/// literal is the only thing standing between "reload the gateway" and -/// "reload anything". +/// The unit name is the scope of this verb: nginx is a host unit, so no +/// namespace bounds it and the literal is the only thing standing between +/// "reload the gateway" and "reload anything". /// /// Returns `(String::new(), String::new())` on success so it fits the /// `exec` return type directly. diff --git a/nix/host-modules/hive-gateway/default.nix b/nix/host-modules/hive-gateway/default.nix index f6858f27..4b4e964a 100644 --- a/nix/host-modules/hive-gateway/default.nix +++ b/nix/host-modules/hive-gateway/default.nix @@ -1,10 +1,9 @@ # Single nginx in front of every hyperhive web surface — dashboard, # per-agent UIs (sub-path), forge + matrix (sub-domain), .well-known # delegations — plus the hive-internal dnsmasq resolver. Both run on the -# HOST, next to hive-c0re. They used to live in a `hive-gateway` -# container that shared the host netns anyway, so the boundary bought no -# network isolation and cost a resolv.conf sync, a reload that had to -# cross the machine bus, and four bind mounts. +# HOST, next to hive-c0re: nginx binds the host's :80/:443 and dnsmasq +# answers on the hive bridge, so neither can be confined to a network +# namespace of its own. # Full vhost map + discovery flow + design rationale in # `docs/gateway.md`. Layout: ./options.nix (option declarations), # ./vhosts.nix (the nginx virtual-host tree), ./error-pages.nix @@ -106,11 +105,9 @@ in ]; # Ensure the gateway state dirs exist at host boot, before anything - # reads or writes them. They used to double as bind-mount sources - # for the container (nspawn would auto-create a missing one); the - # rules stay because they still cover the fresh-boot window before - # c0re has run, and they pin owner + mode rather than leaving it to - # whoever creates the path first. + # reads or writes them: these rules cover the fresh-boot window + # before c0re has run, and pin owner + mode rather than leaving it + # to whoever creates the path first. # # /run/hive-agent — per-agent UDS socket dir, written by c0re's # set_nspawn_flags when agents start. Owned by `hive-core` (the @@ -273,11 +270,8 @@ in # unprivileged and cannot act on a system unit). # # It stays an explicit trigger rather than a systemd path unit - # watching the file. That used to be impossible — an IN_MOVED_TO - # from the atomic rename did not cross the nspawn mount-namespace - # boundary — and with one machine it would now work. It is still - # not wanted: the write and the reload belong in one causal chain - # c0re can retry and report on (see RELOAD_PENDING), not two + # watching the file: the write and the reload belong in one causal + # chain c0re can retry and report on (see RELOAD_PENDING), not two # independent units racing on an inotify event. services.nginx = { @@ -289,10 +283,9 @@ in inherit (nginxTree) appendHttpConfig virtualHosts; }; - # dnsmasq moves with nginx rather than staying behind: it was only in - # the container because nginx was, and leaving it there would keep the - # whole resolv.conf sync machine alive for a resolver that no longer - # needs it. Host-side it reads the one /etc/resolv.conf directly. + # dnsmasq is a host service alongside nginx, so it reads the host's + # /etc/resolv.conf directly and picks up network changes as they + # happen — no copy to keep in sync. services.dnsmasq = import ./dnsmasq.nix { inherit lib diff --git a/swarm-controller/README.md b/swarm-controller/README.md index 2cff09ef..fe96e5f2 100644 --- a/swarm-controller/README.md +++ b/swarm-controller/README.md @@ -42,11 +42,9 @@ What keeps that safe is that the directory holds **one** socket. So: > Pointing nginx at that directory to reach this socket would put the admin > socket within its reach too. -This got *less* forgiving when nginx moved onto the host: the gateway used to -reach a unix upstream through a bind-mount, so the mount list was a second -bound on what it could touch. There is no mount now — the directory is the -whole of the access control. A unit test pins the default path so a tidying -edit fails instead of reviewing cleanly. +nginx is a host service, so nothing narrows what it can reach except the +directory itself — that is the whole of the access control. A unit test pins +the default path so a tidying edit fails instead of reviewing cleanly. `RuntimeDirectoryPreserve=yes` and the daemon's stale-socket unlink on start are a **pair**: preserving the directory without the unlink means `bind` fails with diff --git a/swarm-controller/src/main.rs b/swarm-controller/src/main.rs index 58a16727..ddc14a01 100644 --- a/swarm-controller/src/main.rs +++ b/swarm-controller/src/main.rs @@ -29,10 +29,8 @@ use axum::{Router, routing::get}; /// The directory is its own — deliberately not shared with hive-c0re's /// `/run/hyperhive`. The socket is `0666`, so its directory is the only /// access control it has; co-locating it with c0re's admin socket would -/// put both within reach of whatever can reach either. That used to be -/// enforced by which *directory* was bind-mounted into the gateway -/// container; with nginx on the host the mount is gone and the directory -/// is all that is left, so the rule matters more, not less. +/// put both within reach of whatever can reach either. nginx runs on the +/// host, so nothing narrows its reach for you. const DEFAULT_SOCKET: &str = "/run/swarm-controller/controller.sock"; fn socket_path() -> PathBuf { @@ -98,9 +96,8 @@ mod tests { /// The socket must not share a directory with anything else, because /// the socket is `0666` and the directory is therefore the only access /// control it has. `/run/hyperhive` in particular holds hive-c0re's - /// **admin** socket. nginx used to reach a unix upstream by mounting - /// the socket's whole directory into the gateway container; it runs on - /// the host now, so nothing narrows its reach but the directory itself. + /// **admin** socket, and nginx — a host service — is bounded by + /// nothing but the directory itself. /// /// A test rather than a comment: the failure this guards against is a /// one-word edit that looks tidier and reads fine in review.