From 07852cabc1e80f841e71dc625a8d19a9f32ab8d9 Mon Sep 17 00:00:00 2001 From: atlas Date: Tue, 11 Aug 2026 18:00:27 +0200 Subject: [PATCH] feat(3088): move the gateway's nginx + dnsmasq onto the host MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gateway's nginx + dnsmasq no longer run in their own nspawn container. `nix/host-modules/hive-gateway/default.nix` loses the `containers.hive-gateway` wrapper and everything that existed only to punch holes in it: `privateNetwork = false`, `CAP_NET_ADMIN`, five bind mounts, its own `stateVersion`, `networking.firewall.enable = false`, `networking.resolvconf.enable = false`, and the `hive-gateway-resolv` path+service pair. 465 -> 303 lines. The container never bought isolation here. It shared the host netns by necessity — nginx binds the host's :80/:443, dnsmasq answers on the bridge — so each of those settings was undoing a boundary the gateway could not afford in the first place. Four things made it more than a deletion, none of them visible in the nix diff: - The self-signed cert service also imports the hive CA leaf, so removing it with the container would have left nginx naming a missing cert file, which it refuses to load at all. - The nginx reload is a hive-priv verb. It still needs root, but no longer for the reason its doc gave, and `--machine=` was both transport and scope — so the unit name is now hard-coded in the helper as the containment. - The lifecycle verb named a container that stops existing. - `journalctl -M hive-gateway` had no machine to enter. Per the operator's ruling, the operator verb keeps working and agents lose it. `InfraContainer` answered three questions that used to share an answer; it now splits into `name()` (identity), `target()` (Container vs HostUnit), `service_unit()` (the systemd unit), and `agent_restartable()`, which the MCP restart path checks before the capability so the refusal cannot read as "ask for infra_admin". `SIBLING_CONTAINERS` drops the gateway — it gates the requests that name a container as a string — while `FromStr` still accepts it, because that answers what a name is, not who may act on it. The dashboard's gateway journal reads host journald filtered to `nginx.service`. Prose was corrected where it only named a location, and re-argued where the container was doing security work: a `0666` per-agent socket was safe because only the gateway container had the directory bind-mounted. There is no mount now, so the directory permissions are the whole of the access control — the constraint holds, its mechanism doesn't. Gate: nix fmt / clippy --all-targets -D warnings / cargo test all clean (710 tests); hivectl-cli.md regenerated from the clap tree. The nix eval was run in both TLS shapes at this commit: every delta in the rendered virtualHosts is one of the three intended path moves, dnsmasq settings are byte-identical, and the absence probe flips true -> false with bindMounts emptied. --- docs/boundary.md | 21 +- docs/conventions.md | 2 +- docs/coordinator.md | 6 +- docs/gateway.md | 53 +- docs/network.md | 77 ++- docs/security.md | 18 +- docs/tools/hivectl-cli.md | 6 +- docs/tools/scheduling.md | 6 +- docs/web-ui/dashboard.md | 18 +- hive-agent-mcp/src/mcp/args.rs | 3 +- hive-agent-mcp/src/mcp/mod.rs | 8 +- hive-agent/src/mcp_config.rs | 4 +- hive-agent/src/web_ui/mod.rs | 11 +- hive-c0re/src/dashboard/infra_containers.rs | 20 +- hive-c0re/src/dashboard/journal.rs | 83 ++- hive-c0re/src/dashboard/state_snapshot.rs | 2 +- hive-c0re/src/gateway_nginx.rs | 39 +- hive-c0re/src/lifecycle/mod.rs | 14 +- hive-c0re/src/meta.rs | 6 +- hive-c0re/src/paths.rs | 11 +- hive-c0re/src/priv_client.rs | 19 +- hive-c0re/src/server.rs | 6 +- .../src/socket_server/lifecycle_handlers.rs | 31 +- hive-c0re/src/workers/agent_sockets.rs | 11 +- hive-host-sock/src/lib.rs | 5 +- hive-priv-sock/src/lib.rs | 178 ++++-- hive-priv/src/main.rs | 58 +- hive-sh4re/src/permissions.rs | 10 +- hivectl/src/cli.rs | 2 +- nix/host-modules/hive-gateway/default.nix | 531 +++++++----------- nix/host-modules/hive-gateway/options.nix | 10 +- nix/host-modules/hive-gateway/vhosts.nix | 11 +- swarm-controller/README.md | 15 +- swarm-controller/src/main.rs | 17 +- 34 files changed, 699 insertions(+), 613 deletions(-) diff --git a/docs/boundary.md b/docs/boundary.md index e5093371..fb065c42 100644 --- a/docs/boundary.md +++ b/docs/boundary.md @@ -121,8 +121,25 @@ The mode is load-bearing, not cosmetic. Write permission on a them, and the sticky bit is the only thing that would restrain that (it is not set here). A world-writable socket dir therefore lets anything able to reach the path delete an agent's socket and bind its own — and -the gateway container has all of `/run/hive-agent` bind-mounted in. -Dropping `o=w` removes that permission rather than qualifying it. +nginx reaches all of `/run/hive-agent` (as a plain host path since the +gateway moved out of its container; it used to be bind-mounted in, which +was the same reach through a longer route). Dropping `o=w` removes that +permission rather than qualifying it. + +⚠️ **The gateway leaving its container is a deliberate trade, recorded +here so it is not mistaken for an oversight.** nginx and dnsmasq run on +the host next to `hive-c0re` (see `docs/gateway.md`). What was given up +is a *mount/pid* namespace — **not** a network one: that container ran +with `privateNetwork = false` and shared the host's netns, so nginx was +already binding host ports and already reaching `localhost` upstreams. +The boundary bought no network isolation while costing a resolv.conf +sync, a reload that had to cross the machine bus, and three bind mounts. +🔑 It did cost one real thing, and the replacement is explicit: the +privileged reload verb used to be scoped by `--machine=hive-gateway`, +which could only ever reach into that one container. With no namespace +to bound it, the unit name is hard-coded in `hive-priv` instead — see +`PrivRequest::ReloadGatewayNginx`. **A caller cannot name the unit, so +the verb cannot be steered at another service.** ⚠️ Contrast `/shared`, which *is* sticky world-writable (`1777`): it has many legitimate writers, so sticky is the best available answer there. diff --git a/docs/conventions.md b/docs/conventions.md index 44839b1c..c260d7a6 100644 --- a/docs/conventions.md +++ b/docs/conventions.md @@ -381,7 +381,7 @@ that allows the underlying resource access. | `manage_root_agent` | may lifecycle-manage the root/manager agent via `kill`/`start`/`restart` | | `read_host_journal` | `get_host_journal` MCP tool is registered + `GET /journal-host` requests are served | | `query_agent_state` | may call `get_loose_ends` / `CountPendingReminders` targeting non-child agents | -| `infra_admin` | may call `restart(name)` on hive infrastructure containers (`hive-ci`, `hive-gateway`, `hive-forge`); each restart is logged to the dashboard AUDIT trail | +| `infra_admin` | may call `restart(name)` on hive infrastructure containers (`hive-ci`, `hive-forge`, `hive-matrix` — **not** `hive-gateway`, which is the host's nginx and is operator-only); each restart is logged to the dashboard AUDIT trail | **Config storage** — per-agent capabilities live in `/var/lib/hyperhive/meta/capabilities.json` alongside `tool-groups.json`. diff --git a/docs/coordinator.md b/docs/coordinator.md index b4e6b4c2..047296c0 100644 --- a/docs/coordinator.md +++ b/docs/coordinator.md @@ -500,9 +500,9 @@ Two things to know about the weights: override in `meta/resource-limits.json`, so every agent carries the same value and the weight does *not* rank agents against each other. What `80` buys is that agents yield to everything **not** on this - drop-in path: host services and the infra containers (`hive-ci`, - `hive-forge`, `hive-gateway`, `hive-matrix`), which stay at the - kernel default of `100`. + drop-in path: host services (nginx and dnsmasq among them) and the + infra containers (`hive-ci`, `hive-forge`, `hive-matrix`), which stay + at the kernel default of `100`. - `IOWeight=` is only honoured when the backing device runs the BFQ scheduler or has blk-iocost QoS enabled. On a host using `none`/`mq-deadline`/`kyber` without iocost, systemd writes the value diff --git a/docs/gateway.md b/docs/gateway.md index c0c582ba..8d5f8a66 100644 --- a/docs/gateway.md +++ b/docs/gateway.md @@ -1,6 +1,6 @@ # hive-gateway -Single nginx in front of every hyperhive web surface. Container `hive-gateway`, shared host netns, system-config (not meta-flake managed). Configured via `services.hyperhive.gateway.*` + per-subsystem opt-in flags in `services.hyperhive.{forge,matrix,...}`. +Single nginx in front of every hyperhive web surface. Runs on the **host**, next to hive-c0re; system-config (not meta-flake managed). Configured via `services.hyperhive.gateway.*` + per-subsystem opt-in flags in `services.hyperhive.{forge,matrix,...}`. (It lived in a `hive-gateway` container until #3088 — one that shared the host netns anyway, so the boundary gave no network isolation while costing a resolv.conf sync, a machine-bus reload and three bind mounts.) ## Vhost map @@ -112,18 +112,22 @@ now set unconditionally for every agent. The mechanism: a UDS upstream (`http://unix:/run/hive-agent//web.sock:/`); if the socket is not yet bound, nginx returns 502 caught by the `error_page 502 503 504 = /__hive_agent_unreachable` directive. - The gateway container bind-mounts `/var/lib/hyperhive/gateway/` at - `/run/hive-state/`; nginx includes `/run/hive-state/agents.conf`. - After each write, c0re triggers the appropriate nginx action inside - the gateway container via `hive-priv` (which runs as root and has - `--machine=hive-gateway` transport rights that hive-c0re lacks). + nginx includes `/var/lib/hyperhive/gateway/agents.conf` — the same + path c0re writes, since both run on the host. + After each write, c0re triggers the appropriate nginx action via + `hive-priv` (which is root; hive-c0re runs as the unprivileged + `hive-core` user and cannot act on a system unit). `hive-priv` queries `ActiveState` and dispatches: - active → `systemctl reload nginx` (SIGHUP, zero-downtime) - failed → `systemctl reset-failed nginx` + `systemctl start nginx` - otherwise → `systemctl start nginx` - This is intentionally host-side: `IN_MOVED_TO` from an atomic rename - does not propagate across the nspawn mount-namespace boundary, so a - path unit inside the container would never fire. + This is an explicit trigger rather than a path unit watching the + file. It used to be *impossible* to do it any other way — `IN_MOVED_TO` + from the atomic rename did not cross the nspawn mount-namespace + boundary, so an in-container path unit never fired. With nginx on the + host a path unit would now work, and it is still not wanted: the write + and the reload belong in one causal chain c0re can retry and report on + (`RELOAD_PENDING`), not two units racing on an inotify event. c0re regenerates `agents.conf` (and triggers a reload) on two triggers: every topology change (new/removed agents) and every 10s @@ -160,8 +164,8 @@ selected by which (if any) external TLS source is set: | mode | config | cert source | `.well-known` scheme | |---|---|---|---| | self-signed (default) | neither `tls.certDir` nor `tls.acme` set | host hive-CA signs a gateway leaf (RSA-4096) | `https` | -| ACME (Let's Encrypt) | `tls.acme.enable = true` | nginx inside container via HTTP-01 | `https` | -| operator cert | `tls.certDir` set | bind-mounted from host | `https` | +| ACME (Let's Encrypt) | `tls.acme.enable = true` | nginx via HTTP-01 | `https` | +| operator cert | `tls.certDir` set | read from the operator's dir | `https` | The `gateway.selfSignedTls` option is **deprecated and ignored** — self-signed is now derived from the absence of `tls.certDir` / `tls.acme`. Setting it to @@ -182,7 +186,7 @@ services.hyperhive.gateway = { }; ``` -nginx inside the gateway container obtains and auto-renews certs via the ACME HTTP-01 challenge on `port` (default 80). The gateway container shares the host network namespace (`privateNetwork = false`) so outbound ACME requests work without any extra routing. Certs are stored inside the container's persistent state dir (`/var/lib/acme/` inside `hive-gateway`; survives restarts because `ephemeral = false`). +nginx obtains and auto-renews certs via the ACME HTTP-01 challenge on `port` (default 80). Certs land in `/var/lib/acme/` on the host, managed by nixpkgs's `security.acme` in the ordinary way. **Requirements**: `services.hyperhive.domain` must be publicly DNS-resolvable to this host, and `openFirewall = true` so Let's Encrypt can reach `/.well-known/acme-challenge/`. Each active vhost (main domain, `forge.`, `chat.`) gets its own cert via separate ACME challenges — the swarm services default to names under `services.hyperhive.swarm.domain`, so **every one of those names must resolve to this host too**, not just the hive's own. @@ -194,9 +198,11 @@ Mutual exclusion: `tls.certDir` set together with `tls.acme.enable = true` fails On by default, and listens on `httpsPort` (default 443) on every vhost beside the plain-http `port` (default 80). -The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host service (`hive-tls-ca.service`, from the `hive-tls` module) generates a long-lived CA (`services.hyperhive.tls.caValidityDays`, default ~20y) under `services.hyperhive.tls.stateDir` (default `/var/lib/hive-tls`), then signs a gateway **leaf** (`leafValidityDays`, default 30d) with it. The leaf dir is bind-mounted read-only into the gateway container at `/run/hive-ca`; an in-container import unit copies the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`) with the owner/mode nginx needs. +The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host service (`hive-tls-ca.service`, from the `hive-tls` module) generates a long-lived CA (`services.hyperhive.tls.caValidityDays`, default ~20y) under `services.hyperhive.tls.stateDir` (default `/var/lib/hive-tls`), then signs a gateway **leaf** (`leafValidityDays`, default 30d) with it. `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`). -**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated, in-container leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them. +⚠️ **That import unit is not a leftover of the old container — do not collapse it into pointing nginx at the CA dir.** It does two jobs. It re-modes the leaf (`hive-tls-ca` writes the key `0600 root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a `0600` key fails the config test and blocks the unit), and it guarantees **every cert path the nginx config names exists** — which is what the swarm-services fallback below is for. Removing it re-creates the #3097 outage. + +**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them. **What consumers trust**: `trust-bundle.pem` in the same state dir, not `ca.pem`. The hive CA is itself issued under the swarm root ([`swarm/ca.md`](swarm/ca.md) has the hierarchy), and an intermediate is not a chain a verifier can terminate at — so the bundle carries the hive CA plus whatever it is rooted at. nginx is handed the leaf with the hive CA appended for the same reason. Everything that trusts the hive's TLS reads the bundle: agents (via `security.pki.certificateFiles`), the CI and forge containers, and a federating peer. @@ -220,11 +226,11 @@ services.hyperhive.gateway = { }; ``` -The directory is bind-mounted read-only into the gateway container at `/run/hive-tls/`. nginx uses `cert.pem` + `key.pem` (override `tls.certName`/`tls.keyName` for different filenames). Both modes listen on `httpsPort` (default 443) and emit `https://` in `.well-known` responses. +nginx reads the directory directly and uses `cert.pem` + `key.pem` (override `tls.certName`/`tls.keyName` for different filenames). Both modes listen on `httpsPort` (default 443) and emit `https://` in `.well-known` responses. `tls.certDir` and `tls.acme.enable` set together is an assertion error. -**Key file permissions**: nixpkgs's `security.acme` outputs private keys as `0640 root:acme` by default. nginx inside the gateway container runs as the `nginx` user and cannot read a key with that ownership. Fix with: +**Key file permissions**: nixpkgs's `security.acme` outputs private keys as `0640 root:acme` by default. nginx runs as the `nginx` user and cannot read a key with that ownership. Fix with: ```nix security.acme.certs."example.com".group = "nginx"; @@ -424,11 +430,12 @@ JS/CSS assets skip `try_files` entirely and get the immutable cache headers. Nix store paths are content-addressed — the hash changes on any content change — so `max-age=31536000` is safe. -**Why nix store is reachable from the gateway container**: nspawn -containers bind-mount `/nix/store` read-only by default. The -`HIVE_AGENT_FRONTEND_DIR` path is a nix store path baked in at -hive-c0re build time — the same path is visible to both c0re (writing -`agents.conf`) and the gateway nginx (serving files from it). +**Why the nix store path resolves**: `HIVE_AGENT_FRONTEND_DIR` is a nix +store path baked in at hive-c0re build time, and c0re (writing +`agents.conf`) and nginx (serving files from it) are on the same machine, +so they see the same store. This used to need explaining — nspawn +bind-mounts `/nix/store` read-only into a container, which is what made +the baked-in path work from inside the gateway. **Graceful degradation**: if `HIVE_AGENT_FRONTEND_DIR` is empty or unset (e.g. a build that predates `cfg.frontend`), each agent gets the @@ -493,9 +500,7 @@ services.hyperhive.gateway.auth = { The credential store lives at the fixed path `/var/lib/hyperhive/gateway/gateway.htpasswd` on the host. A tmpfiles rule pre-creates the file on first boot; no manual path configuration -is required. The file is exposed inside the gateway container at -`/run/hive-state/gateway.htpasswd` via the existing gateway state -bind-mount. +is required. nginx reads it at that path directly. Manage users with `hivectl gateway`. `hivectl` sends the request over the host admin socket and the `hive-c0re` daemon performs the write at its diff --git a/docs/network.md b/docs/network.md index 54a15428..a64e50ac 100644 --- a/docs/network.md +++ b/docs/network.md @@ -115,49 +115,46 @@ schemes pick their own. dnsmasq is **authoritative** for the hive's own zones — answers ``, `forge.`, `matrix.` queries with the bridge IP (where nginx is reachable). Everything -else is forwarded to the host's own resolvers: dnsmasq reads the -gateway container's `/etc/resolv.conf`, the host copy nixos-container -makes at each container start. Containers don't need to know the -upstream — they query the bridge IP and dnsmasq does the right thing -per-name. +else is forwarded to the host's own resolvers: dnsmasq runs on the host +and reads the host's `/etc/resolv.conf` directly. Containers don't need +to know the upstream — they query the bridge IP and dnsmasq does the +right thing per-name. -That copy is one-shot — systemd-nspawn(1) is explicit that nothing -propagates into it after early init, because resolv.conf is normally -updated by rename rather than in place. Left alone, a host network -change (new router, new lease, laptop moving networks) would strand -dnsmasq on a resolver that no longer answers, and every non-hive -lookup from every agent would hang until someone restarted the -gateway. The host-side **`hive-gateway-resolv`** path unit closes -that: it watches `/etc/resolv.conf`, `machinectl copy-to`s it into -the container, and reloads dnsmasq (`SIGHUP` — re-read upstreams + -flush cache, nothing dropped). The watch is armed before -`network-pre.target` so the boot's first DHCP write is caught as well, -and the sync also runs once per gateway start to pick up a resolver -change that happened while the container was down. A host file with no -`nameserver` line is -skipped rather than pushed, so a mid-rewrite snapshot can't blank the -hive's DNS. There is deliberately no fallback `server=`: dnsmasq -queries all known upstreams in parallel, so a hardcoded public -resolver would take a share of normal traffic, not just cover the gap. +There is deliberately no fallback `server=`: dnsmasq queries all known +upstreams in parallel, so a hardcoded public resolver would take a share +of normal traffic, not just cover the gap. -Two alternatives that look simpler and aren't: +### History: the resolv.conf sync, and why it is gone -- **A path unit inside the container.** The host replaces - `/etc/resolv.conf` by rename, and that `IN_MOVED_TO` doesn't cross - the nspawn mount namespace — the same reason hive-c0re reloads nginx - from the host side after each `agents.conf` write. -- **Bind-mounting the host's `/etc/resolv.conf` into the container.** - openresolv writes a temp file and renames over the target, so the - bind mount would pin the *first* inode for the container's whole - lifetime — strictly worse than the copy, which at least a restart - clears. (Reachability is not the problem here: the gateway runs with - `privateNetwork = false`, so it shares the host's netns and can reach - anything the host can.) +Until the gateway moved onto the host, dnsmasq ran in the `hive-gateway` +container and read *that* container's `/etc/resolv.conf` — a one-shot +copy nixos-container made at start. systemd-nspawn(1) is explicit that +nothing propagates into it after early init, because resolv.conf is +normally updated by rename rather than in place. So a host network change +(new router, new lease, laptop moving networks) stranded dnsmasq on a +resolver that no longer answered, and every non-hive lookup from every +agent hung until someone restarted the gateway. -`machinectl copy-to` is used rather than writing to the container's -rootfs from the host, so the push goes through the container's own -mount namespace and stays correct if `/etc` is ever assembled -differently (e.g. `system.etc.overlay`). +A host-side `hive-gateway-resolv` path unit closed that gap: watch +`/etc/resolv.conf`, `machinectl copy-to` it into the container, reload +dnsmasq. Roughly eighty lines of watcher, marker file, is-active guard +and mid-rewrite-snapshot check — **all of it bridging two copies of one +file.** With one machine there is one file, and the whole unit is +deleted. + +🔑 Worth keeping as a shape, not just a story: **the sync was not +complexity anyone chose. It was the cost of a boundary that bought +nothing here** — the gateway already ran with `privateNetwork = false`, +sharing the host's netns, so the container never provided network +isolation in the first place. When a workaround is that elaborate, the +question to ask is what the boundary is *for*. + +(Two alternatives were considered at the time and both were worse than +the copy: a path unit *inside* the container never fired, because the +host replaces the file by rename and `IN_MOVED_TO` does not cross the +nspawn mount namespace; and bind-mounting the host's `/etc/resolv.conf` +would have pinned the *first* inode for the container's whole lifetime, +since openresolv writes a temp file and renames over the target.) `bind-interfaces` + `interface = [ bridgeName "lo" ]` means the listener only accepts queries from the bridge interface (plus lo for @@ -289,4 +286,4 @@ loopback — the hive-c0re admin socket is a UDS, not TCP. ## Cross-references -- `docs/gateway.md` — vhost map + the gateway container's other duties +- `docs/gateway.md` — vhost map + the gateway's other duties diff --git a/docs/security.md b/docs/security.md index d0ed66d3..bc2f7de4 100644 --- a/docs/security.md +++ b/docs/security.md @@ -187,8 +187,20 @@ bind-mount compatibility with user namespace UID mapping and is tracked as a TOD `hive-c0re` runs as the unprivileged system user `hive-core` (`/var/lib/hyperhive` owned by `hive-core:hive-core`). It cannot -directly invoke `nixos-container`, `journalctl -M`, or `systemctl --M hive-gateway` — those require root. `hive-priv` fills this gap. +directly invoke `nixos-container`, `journalctl -M`, or act on a system +unit (`systemctl reload nginx`) — those require root. `hive-priv` fills +this gap. + +⚠️ **Note what that costs when a helper verb loses its namespace.** +`ReloadGatewayNginx` used to run `systemctl -M hive-gateway …`, and the +`--machine=` flag was doing two jobs: it was the *transport* into the +container **and** the *scope* — the verb could not reach anything +outside that one machine. With nginx on the host the transport is +unnecessary and the scope went with it, so the containment is now the +hard-coded unit name in `hive-priv`: a caller cannot name the unit, so +the verb cannot be steered at another service. **When a privileged verb +stops needing a namespace, check whether the namespace was also what +bounded it.** ### hive-priv @@ -207,7 +219,7 @@ known operations; there is no arbitrary command pass-through: | `DestroyContainer` | `nixos-container destroy ` | | `ListContainers` | `nixos-container list` | | `ReadContainerJournal` | `journalctl -M -n [filters...]` | -| `ReloadGatewayNginx` | `systemctl -M hive-gateway reload/start/reset-failed nginx` | +| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) | | `WriteNspawnFlags` | write `/etc/nixos-containers/.conf` (bind-mount list + network isolation vars) | | `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container | | `RemoveServiceDropin` | remove `container@.service.d/` drop-in on destroy | diff --git a/docs/tools/hivectl-cli.md b/docs/tools/hivectl-cli.md index fc8333c4..6d17ff19 100644 --- a/docs/tools/hivectl-cli.md +++ b/docs/tools/hivectl-cli.md @@ -808,7 +808,7 @@ Bare `hivectl stop` stops everything; scope flags narrow it to specific sub-agen * `--agent ` — A specific sub-agent by name. Repeatable: `--agent a --agent b` * `--ci` — The CI runner container (`hive-ci`) * `--forge` — The forge container (`hive-forge`) -* `--gateway` — The gateway container (`hive-gateway`) +* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container * `--matrix` — The matrix container (`hive-matrix`) * `--graceful` — Gracefully quiesce each agent before stopping, instead of a hard stop. Each agent gets a graceful-stop DAG on the job queue: the harness is signalled, runs one stop-checkpoint turn to flush durable `/state`, drains, then the container is stopped (bounded by a 3-min timeout that falls back to a hard stop). All drains overlap. Applies to agents only * `--no-wait` — Return immediately after the stop DAGs are queued instead of waiting for them with live per-node progress @@ -829,7 +829,7 @@ Bare `hivectl start` restores the agents stopped by the last broad-scope `stop` * `--agent ` — A specific sub-agent by name. Repeatable: `--agent a --agent b` * `--ci` — The CI runner container (`hive-ci`) * `--forge` — The forge container (`hive-forge`) -* `--gateway` — The gateway container (`hive-gateway`) +* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container * `--matrix` — The matrix container (`hive-matrix`) * `--no-wait` — Return immediately after the start DAGs are queued instead of waiting for them with live per-node progress @@ -849,7 +849,7 @@ Bare `hivectl restart` restarts everything; scope flags narrow it. * `--agent ` — A specific sub-agent by name. Repeatable: `--agent a --agent b` * `--ci` — The CI runner container (`hive-ci`) * `--forge` — The forge container (`hive-forge`) -* `--gateway` — The gateway container (`hive-gateway`) +* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container * `--matrix` — The matrix container (`hive-matrix`) * `--graceful` — Gracefully quiesce each agent on the stop half (see `stop --graceful`). Applies to agents only diff --git a/docs/tools/scheduling.md b/docs/tools/scheduling.md index 1b02918c..4a4a90ac 100644 --- a/docs/tools/scheduling.md +++ b/docs/tools/scheduling.md @@ -79,8 +79,10 @@ lifecycle events, or another container's boot log. - `unit` — filter to a systemd unit (e.g. `hive-c0re.service`). - `container` — nspawn machine name verbatim. Agent containers use the `h-` prefix (e.g. `h-iris`); infrastructure containers - use their full name (e.g. `hive-ci`, `hive-forge`, `hive-matrix`, - `hive-gateway`). Omit for the host journal. + use their full name (e.g. `hive-ci`, `hive-forge`, `hive-matrix`). + Omit for the host journal. The gateway has no machine — its nginx + runs on the host, so read it with `unit: nginx.service` and no + `container`. - `lines` — how many lines to return (default 30, max 100). - `priority` — minimum syslog level (`emerg` … `debug`). - `grep` — regex matched against log message fields (`journalctl --grep`). diff --git a/docs/web-ui/dashboard.md b/docs/web-ui/dashboard.md index 7cb7cb40..6183d159 100644 --- a/docs/web-ui/dashboard.md +++ b/docs/web-ui/dashboard.md @@ -180,10 +180,12 @@ omitted — agents share the host netns, so there is no per-container net counter (per-agent network needs the netns-isolation roadmap in `docs/network.md`). -**1NFR4** — start / stop / restart the four hive infrastructure -containers (`hive-ci`, `hive-forge`, `hive-gateway`, `hive-matrix`) -directly from the dashboard, without needing an `infra_admin` agent's -`restart` MCP tool. One row per container: name, a `badge-ok`/`badge-fail` +**1NFR4** — start / stop / restart the four hive infrastructure services +(`hive-ci`, `hive-forge`, `hive-gateway`, `hive-matrix`) directly from the +dashboard, without needing an `infra_admin` agent's `restart` MCP tool. +Three are containers; `hive-gateway` is the host's `nginx.service`, and is +the one an agent may **not** restart — this panel is the way it gets +bounced. One row per service: name, a `badge-ok`/`badge-fail` running/stopped dot, and `↺ R3ST4RT` + `■ ST0P` (running) or `▶ ST4RT` (stopped) buttons, same themed-confirm pattern as the K3PT ST4T3 tombstone actions. Backed by @@ -460,7 +462,7 @@ The current capabilities are: | `manage_root_agent` | allows the `set_status` / lifecycle tools on the root agent | | `read_host_journal` | unlocks `get_host_journal` to read journald from inside a container | | `query_agent_state` | allows `get_loose_ends(agent: "")` calls targeting other agents | -| `infra_admin` | allows `restart` on hive infrastructure containers (`hive-ci`, `hive-gateway`, `hive-forge`); each restart is logged to the AUDIT trail | +| `infra_admin` | allows `restart` on hive infrastructure containers (`hive-ci`, `hive-forge`, `hive-matrix`; the gateway is operator-only); each restart is logged to the AUDIT trail | Each row is one agent. Columns are the capability names returned by `GET /api/capabilities` as `caps: Vec`. Checking or unchecking @@ -1340,9 +1342,11 @@ below — some endpoints aren't in it yet. rootfs every ~5 min, `-x` excluding the shared read-only nix store. `null` until the first sample lands. - `POST /api/infra-container/{name}/{action}` — start / stop / restart a - hive infra container (C0R3 › 1NFR4 panel). `name` parses into the + hive infra service (C0R3 › 1NFR4 panel). `name` parses into the `InfraContainer` allowlist (`hive-ci`/`hive-forge`/`hive-gateway`/ - `hive-matrix`, 400 on unknown), `action` ∈ `start|stop|restart`. Calls + `hive-matrix`, 400 on unknown), and the variant decides the unit — + `container@.service`, or `nginx.service` for the gateway. + `action` ∈ `start|stop|restart`. Calls the same `priv_client::control_infra_container` helper the `infra_admin` agent path uses; records an `audit_log` entry (`start_infra`/`stop_infra`/`restart_infra`, actor `"operator"`) either diff --git a/hive-agent-mcp/src/mcp/args.rs b/hive-agent-mcp/src/mcp/args.rs index 260bfec5..6a3f5442 100644 --- a/hive-agent-mcp/src/mcp/args.rs +++ b/hive-agent-mcp/src/mcp/args.rs @@ -326,7 +326,8 @@ pub struct GetHostJournalArgs { /// nspawn machine name verbatim (e.g. `h-iris`). Omit for host journal. /// Agent containers use the `h-` prefix (e.g. `h-iris`); infrastructure /// containers use their full name (e.g. `hive-ci`, `hive-forge`, - /// `hive-matrix`, `hive-gateway`). + /// `hive-matrix`). The gateway has no machine — its nginx runs on the + /// host, so read it with `unit: nginx.service` and no `container`. #[serde(default)] pub container: Option, /// Number of lines to return (default 30, max 100). diff --git a/hive-agent-mcp/src/mcp/mod.rs b/hive-agent-mcp/src/mcp/mod.rs index 96ecf1bd..4366e59a 100644 --- a/hive-agent-mcp/src/mcp/mod.rs +++ b/hive-agent-mcp/src/mcp/mod.rs @@ -681,8 +681,9 @@ impl AgentServer { Only succeeds if `name` is a direct child of this agent in the topology \ tree — the server enforces this. No approval required. \ Agents holding the `infra_admin` capability may also pass a hive \ - infrastructure container name (`hive-ci`, `hive-gateway`, `hive-forge`) \ - to restart it directly via the privileged helper." + infrastructure container name (`hive-ci`, `hive-forge`, `hive-matrix`) \ + to restart it directly via the privileged helper. The gateway is \ + not restartable by an agent — ask the operator." )] async fn restart(&self, Parameters(args): Parameters) -> String { let log = format!("{args:?}"); @@ -790,7 +791,8 @@ impl AgentServer { `container`: nspawn machine name verbatim (e.g. `h-iris`). Omit for host journal. \ Agent containers use the `h-` prefix (e.g. `h-iris`, `h-atlas`); \ infrastructure containers use their full name (e.g. `hive-ci`, `hive-forge`, \ - `hive-matrix`, `hive-gateway`). \ + `hive-matrix`). The gateway is not a container: its nginx logs are in \ + the host journal, so omit `container` and pass `unit: nginx.service`. \ `lines`: how many lines (default 30, max 100). \ `priority`: minimum syslog level enum. \ `grep`: regex matched against log message fields (journalctl --grep). \ diff --git a/hive-agent/src/mcp_config.rs b/hive-agent/src/mcp_config.rs index ec23b8d6..fb7b92cc 100644 --- a/hive-agent/src/mcp_config.rs +++ b/hive-agent/src/mcp_config.rs @@ -60,8 +60,8 @@ fn allowed_capability_tools() -> Vec { match t.as_str() { "read_host_journal" => tools.push("get_host_journal".to_owned()), // infra_admin lets an agent restart hive infrastructure - // containers (hive-ci / hive-gateway / hive-forge) through the - // existing `restart` tool. Unlock it here so agents that hold + // containers (hive-ci / hive-forge / hive-matrix — not the + // gateway) through the existing `restart` tool. Unlock it here so agents that hold // the capability without the full `lifecycle` group can still // call it; c0re re-checks the capability server-side and only // honours infra-container names via this path. diff --git a/hive-agent/src/web_ui/mod.rs b/hive-agent/src/web_ui/mod.rs index 13d820be..05a5badc 100644 --- a/hive-agent/src/web_ui/mod.rs +++ b/hive-agent/src/web_ui/mod.rs @@ -169,11 +169,12 @@ pub async fn serve( /// Best-effort unlinks any stale socket left from a crashed previous /// harness (clean exit removes it, but `bind(2)` refuses to overwrite /// an existing file) and `mkdir -p`s the parent for first-boot. Mode -/// `0o666` — world-accessible so the gateway container's nginx process -/// can `connect(2)` without sharing a group with the agent user. -/// The per-agent subdir (`/run/hive-agent//`) is only accessible -/// to containers that have it bind-mounted, so world-accessible sockets -/// are not a material risk. +/// `0o666` — world-accessible so the gateway's nginx process can +/// `connect(2)` without sharing a group with the agent user. What bounds +/// that is the per-agent subdir (`/run/hive-agent//`): the socket +/// mode grants everyone, the directory decides who gets to ask. It used +/// to be bind-mounted into the one container that needed it; nginx is a +/// host unit now, so the directory's own permissions are the whole story. /// /// Marker-gating + the gateway-side consumer: see /// [`docs/gateway.md::Per-agent unix-socket upstream`](../../../docs/gateway.md). diff --git a/hive-c0re/src/dashboard/infra_containers.rs b/hive-c0re/src/dashboard/infra_containers.rs index de446372..45951b09 100644 --- a/hive-c0re/src/dashboard/infra_containers.rs +++ b/hive-c0re/src/dashboard/infra_containers.rs @@ -1,9 +1,13 @@ -//! Dashboard endpoint for operator-driven infra-container lifecycle -//! (start / stop / restart on `hive-ci`, `hive-forge`, `hive-gateway`, -//! `hive-matrix`). Parallels the `infra_admin`-gated agent path in +//! Dashboard endpoint for operator-driven infra lifecycle (start / stop / +//! restart on `hive-ci`, `hive-forge`, `hive-gateway`, `hive-matrix`). +//! Parallels the `infra_admin`-gated agent path in //! `socket_server/lifecycle_handlers.rs::handle_restart_infra`, but this one //! is reached from the dashboard — already fully operator-authenticated — //! so no capability check is needed here, just the same audit trail. +//! +//! The two surfaces no longer cover the same set: the gateway is the +//! operator's to restart and not an agent's, since nginx on the host fronts +//! every hive service. This endpoint keeps all four. use axum::{ extract::{Path as AxumPath, State}, @@ -27,7 +31,7 @@ use super::{AppState, error_response}; post, path = "/api/infra-container/{name}/{action}", params( - ("name" = String, Path, description = "infra container name (hive-ci/hive-forge/hive-gateway/hive-matrix)"), + ("name" = String, Path, description = "infra service name (hive-ci/hive-forge/hive-gateway/hive-matrix)"), ("action" = String, Path, description = "start | stop | restart"), ), responses( @@ -53,8 +57,8 @@ pub(super) async fn post_infra_container( )); } }; - let unit = container.unit_name(); - tracing::info!(%unit, %action, "dashboard: infra container action"); + let target = container.name(); + tracing::info!(%target, %action, "dashboard: infra container action"); let result = crate::priv_client::control_infra_container(container, infra_action).await; let outcome = if result.is_ok() { crate::audit_log::AuditOutcome::Ok @@ -66,12 +70,12 @@ pub(super) async fn post_infra_container( state .coord .audit_log - .record("operator", action_label, unit, outcome, detail.as_deref()) + .record("operator", action_label, target, outcome, detail.as_deref()) { state.coord.emit_audit_entry(entry); } match result { Ok(()) => (StatusCode::OK, "ok").into_response(), - Err(e) => error_response(&format!("{unit}: {e:#}")), + Err(e) => error_response(&format!("{target}: {e:#}")), } } diff --git a/hive-c0re/src/dashboard/journal.rs b/hive-c0re/src/dashboard/journal.rs index 6160b47b..dfdd02b3 100644 --- a/hive-c0re/src/dashboard/journal.rs +++ b/hive-c0re/src/dashboard/journal.rs @@ -1,13 +1,15 @@ //! Journal-read endpoints for the dashboard. //! //! `GET /api/journal/{name}` reads a managed agent container's journal, OR -//! one of the four hive infra containers (`hive-ci`, `hive-forge`, +//! one of the four hive infra services (`hive-ci`, `hive-forge`, //! `hive-gateway`, `hive-matrix` — [`hive_priv_sock::InfraContainer`] is the -//! allowlist), via the root helper (`journalctl -M`, delegated to hive-priv -//! since hive-c0re is unprivileged). `GET /api/journal-host` reads -//! host-side journald, both gated by an allow-list of known units so -//! arbitrary unit names can't be probed. Operator-only by virtue of the -//! dashboard binding host-only. +//! allowlist). A container's journal is a `journalctl -M` read, delegated +//! to the root helper since entering a machine needs privileges hive-c0re +//! doesn't have; `hive-gateway` is nginx on the host, so it reads host +//! journald filtered to that unit and needs no helper at all. +//! `GET /api/journal-host` reads host-side journald, both gated by an +//! allow-list of known units so arbitrary unit names can't be probed. +//! Operator-only by virtue of the dashboard binding host-only. use axum::{ extract::Path as AxumPath, @@ -40,11 +42,12 @@ pub(super) struct JournalQuery { /// container namespace and needs root — is delegated to hive-priv. /// /// `name` is either a managed agent name (`iris`, optionally already -/// carrying the `h-` prefix) or one of the four infra container names -/// (`hive-ci` / `hive-forge` / `hive-gateway` / `hive-matrix` — see -/// [`hive_priv_sock::InfraContainer`]). Infra containers don't run the -/// per-agent hive daemons, so `unit` is ignored for them — always the -/// full machine journal. +/// carrying the `h-` prefix) or one of the four infra names (`hive-ci` / +/// `hive-forge` / `hive-gateway` / `hive-matrix` — see +/// [`hive_priv_sock::InfraContainer`]). Infra targets don't run the +/// per-agent hive daemons, so `unit` is ignored for them — the whole +/// machine journal, or for the gateway the host journal filtered to its +/// own unit. #[utoipa::path( get, path = "/api/journal/{name}", @@ -66,7 +69,17 @@ pub(super) async fn get_journal( let lines = q.lines.unwrap_or(500).min(5000); if let Ok(infra) = name.parse::() { - return read_journal_response(infra.unit_name(), None, lines).await; + return match infra.target() { + hive_priv_sock::InfraTarget::Container(machine) => { + read_journal_response(machine, None, lines).await + } + // No machine to enter — the gateway's nginx is a host unit, so + // this is a plain host-journal read filtered to it. `-M` is + // what needed root here, not journalctl itself. + hive_priv_sock::InfraTarget::HostUnit(unit) => { + read_host_journal_response(Some(unit), lines).await + } + }; } // Defense-in-depth format check so weird chars never reach the @@ -180,21 +193,43 @@ pub(super) async fn get_journal_host( axum::extract::Query(q): axum::extract::Query, ) -> Result { let lines = q.lines.unwrap_or(500).min(5000); - let allowed = ["hive-c0re.service", "hive-priv.service"]; + // `nginx.service` is the gateway: its logs used to live in the + // hive-gateway container's journal and are host-side now. + let allowed = ["hive-c0re.service", "hive-priv.service", "nginx.service"]; + let unit = match q.unit.as_deref().filter(|s| !s.is_empty()) { + Some(u) => { + let unit = if u.ends_with(".service") { + u.to_owned() + } else { + format!("{u}.service") + }; + if !allowed.contains(&unit.as_str()) { + return Err(ProblemDetails::from_status_code(StatusCode::BAD_REQUEST) + .with_detail(format!("journal-host: unknown unit {unit:?}"))); + } + Some(unit) + } + None => None, + }; + read_host_journal_response(unit.as_deref(), lines).await +} + +/// `journalctl [-u ]` on the host + response formatting. No `-M`, so +/// no root and no priv-client hop — hive-c0re reads host journald directly. +/// +/// ⚠️ `unit` is trusted by the time it gets here: [`get_journal_host`] +/// allow-lists an operator-supplied one, and [`get_journal`] passes a unit +/// that came from the [`hive_priv_sock::InfraContainer`] enum. Don't hand +/// this a raw query parameter. +async fn read_host_journal_response( + unit: Option<&str>, + lines: u32, +) -> Result { let mut cmd = tokio::process::Command::new("journalctl"); cmd.args(["--no-pager", "--output=short-iso", "--lines"]) .arg(lines.to_string()); - if let Some(u) = q.unit.as_deref().filter(|s| !s.is_empty()) { - let unit = if u.ends_with(".service") { - u.to_owned() - } else { - format!("{u}.service") - }; - if !allowed.contains(&unit.as_str()) { - return Err(ProblemDetails::from_status_code(StatusCode::BAD_REQUEST) - .with_detail(format!("journal-host: unknown unit {unit:?}"))); - } - cmd.args(["-u", &unit]); + if let Some(u) = unit { + cmd.args(["-u", u]); } match cmd.output().await { Ok(out) => { diff --git a/hive-c0re/src/dashboard/state_snapshot.rs b/hive-c0re/src/dashboard/state_snapshot.rs index 1726a4ec..d3d4bc08 100644 --- a/hive-c0re/src/dashboard/state_snapshot.rs +++ b/hive-c0re/src/dashboard/state_snapshot.rs @@ -154,7 +154,7 @@ async fn infra_container_views() -> Vec { let mut infra_containers = Vec::with_capacity(hive_priv_sock::InfraContainer::ALL.len()); for container in hive_priv_sock::InfraContainer::ALL { infra_containers.push(InfraContainerView { - name: container.unit_name(), + name: container.name(), running: crate::lifecycle::infra_is_running(container).await, }); } diff --git a/hive-c0re/src/gateway_nginx.rs b/hive-c0re/src/gateway_nginx.rs index 69fffa9e..e4cbc694 100644 --- a/hive-c0re/src/gateway_nginx.rs +++ b/hive-c0re/src/gateway_nginx.rs @@ -1,8 +1,7 @@ //! Runtime nginx include-file generator for the gateway's per-agent //! `/agent//` location blocks. Writes //! `/var/lib/hyperhive/gateway/agents.conf` on every topology change. -//! UDS upstream selection, reload trigger (`systemd-run -//! --machine=hive-gateway`), and idempotency: +//! UDS upstream selection, the reload trigger, and idempotency: //! `docs/gateway.md::Per-agent unix-socket upstream`. use anyhow::{Context, Result}; @@ -19,13 +18,13 @@ use crate::agent_sockets; /// Set when `write` publishes a new agents.conf; cleared when /// `reload_gateway_nginx` submits the reload command successfully. /// Lets `spawn_poll` retry the reload on subsequent ticks when the -/// previous attempt failed (e.g. gateway container temporarily down, -/// systemd-run not found) without re-writing the already-correct file. +/// previous attempt failed (e.g. gateway nginx temporarily down, priv +/// helper unreachable) without re-writing the already-correct file. static RELOAD_PENDING: AtomicBool = AtomicBool::new(false); /// Unix timestamp (seconds) of the last failed reload attempt. /// `reload_if_pending` backs off to once per `RELOAD_RETRY_SECS` after a -/// failure so a permanently broken gateway (bad config, container down) +/// failure so a permanently broken gateway (bad config, nginx down) /// doesn't hammer `systemctl` on every 10-second `spawn_poll` tick. static LAST_FAILED_RELOAD: AtomicU64 = AtomicU64::new(0); @@ -79,7 +78,7 @@ fn render(names: &[String], frontend_dir: Option<&str>) -> String { let mut out = String::from( "# Generated by hive-c0re \u{2014} do not edit.\ \n# Refreshed on every topology change + when agents bind/drop their unix sockets.\ - \n# Reload triggered by hive-c0re via systemd-run --machine=hive-gateway.\n", + \n# Reload triggered by hive-c0re via hive-priv (systemctl reload nginx).\n", ); for name in names { // Two upstream forms because named locations (split mode's @@ -165,15 +164,19 @@ fn render(names: &[String], frontend_dir: Option<&str>) -> String { /// body matches what's already on disk (idempotent; avoids spurious /// gateway reloads on a quiet tick). /// -/// After a successful write, triggers the appropriate nginx action inside -/// the gateway container via `hive-priv` (which has the -/// `--machine=hive-gateway` transport rights hive-c0re lacks): -/// reload when nginx is active, reset-failed+start when in a failed -/// state, plain start otherwise. This is intentionally host-side rather -/// than relying on a systemd path unit inside the container watching the -/// bind-mounted file: `IN_MOVED_TO` (fired by the atomic rename) does -/// not reliably propagate across the nspawn mount-namespace boundary, so -/// the path-unit approach was silently broken (see `docs/gateway.md`). +/// After a successful write, triggers the appropriate nginx action via +/// `hive-priv` (hive-c0re runs unprivileged and cannot act on a system +/// unit): reload when nginx is active, reset-failed+start when in a +/// failed state, plain start otherwise. Writer and nginx are now on the +/// same machine, so this is a plain unit action rather than the old +/// `systemd-run --machine=hive-gateway` hop across the container +/// boundary. It stays an explicit trigger rather than a systemd path +/// unit watching the file. A path unit would now *work* — `IN_MOVED_TO` +/// (fired by the atomic rename) failed to propagate across the nspawn +/// mount-namespace boundary, and that boundary is gone — but it is still +/// not wanted: the write already knows it changed something, and a +/// watcher turns one causal edge into a race with the writer's own +/// rename (see `docs/gateway.md`). /// /// The priv call is best-effort — a failed sync is logged but not fatal. /// `reload_if_pending` retries on the next `spawn_poll` tick so a @@ -212,7 +215,7 @@ pub async fn write(names: &[String]) -> Result<()> { /// Retry a pending nginx reload if a previous attempt failed. /// Called by `spawn_poll` on each tick so a transient failure -/// (gateway container temporarily down, systemd-run error) is +/// (gateway nginx temporarily down, priv-helper error) is /// recovered automatically without requiring a new file write. /// /// Backs off to one retry per `RELOAD_RETRY_SECS` after a failure so a @@ -240,8 +243,8 @@ pub async fn reload_if_pending() { /// Synchronise the gateway nginx unit with the current agents.conf via /// `hive-priv` (privileged helper). The state-aware logic (active → /// reload; failed → reset-failed + start; inactive/unknown → start) -/// runs inside hive-priv where it has the `--machine=hive-gateway` -/// transport rights that hive-c0re (unprivileged) lacks. +/// runs inside hive-priv, which is root; hive-c0re runs as the +/// unprivileged `hive-core` user and cannot act on a system unit. /// /// `RELOAD_PENDING` is cleared only after a successful operation so /// `reload_if_pending` keeps retrying on failure. diff --git a/hive-c0re/src/lifecycle/mod.rs b/hive-c0re/src/lifecycle/mod.rs index 4f681dad..a93edadd 100644 --- a/hive-c0re/src/lifecycle/mod.rs +++ b/hive-c0re/src/lifecycle/mod.rs @@ -604,14 +604,14 @@ pub async fn is_running(name: &str) -> bool { .is_ok_and(|s| s.success()) } -/// True when a hive infrastructure container's systemd unit is active. -/// Sibling of [`is_running`] for sub-agents, but infra container/unit names -/// (`hive-ci`, …) already have no `h-` prefix to strip, so this queries -/// `container@.service` directly rather than going through -/// [`container_name`]. Used by the dashboard C0R3 page's 1NFR4 sub-tab to -/// show each infra container's live status dot. +/// True when a hive infrastructure service's systemd unit is active. +/// Sibling of [`is_running`] for sub-agents, but infra names (`hive-ci`, …) +/// have no `h-` prefix to strip and are not all containers, so the unit +/// comes from the variant itself rather than from [`container_name`]. Used +/// by the dashboard C0R3 page's 1NFR4 sub-tab to show each one's live +/// status dot. pub async fn infra_is_running(container: hive_priv_sock::InfraContainer) -> bool { - let unit = format!("container@{}.service", container.unit_name()); + let unit = container.service_unit(); Command::new("systemctl") .args(["is-active", "--quiet", &unit]) .status() diff --git a/hive-c0re/src/meta.rs b/hive-c0re/src/meta.rs index ae2909cd..54d3e6ef 100644 --- a/hive-c0re/src/meta.rs +++ b/hive-c0re/src/meta.rs @@ -168,9 +168,9 @@ pub async fn sync_agents(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<()> { } // Refresh /var/lib/hyperhive/gateway/agents.conf — the nginx include - // file the gateway container bind-mounts and nginx reads at runtime. - // c0re triggers a reload (or start) inside hive-gateway via hive-priv - // after writing the file. Same best-effort + non-fatal shape. + // file the gateway reads at runtime. c0re then triggers a reload (or + // start) of the host's nginx via hive-priv, since c0re is unprivileged. + // Same best-effort + non-fatal shape. if let Err(e) = crate::gateway_nginx::write(&agent_names).await { tracing::warn!(error = ?e, "gateway_nginx::write failed (non-fatal)"); } diff --git a/hive-c0re/src/paths.rs b/hive-c0re/src/paths.rs index b98c2c64..0d26d019 100644 --- a/hive-c0re/src/paths.rs +++ b/hive-c0re/src/paths.rs @@ -219,11 +219,12 @@ pub fn shared_root() -> PathBuf { pub const KNOWLEDGE_DIR: &str = "/var/lib/hyperhive/knowledge"; /// `gateway/` — generated nginx include fragments for the gateway vhost. -/// The gateway container bind-mounts *this subdir only* (not the whole -/// state root) at `/run/hive-state/`, so nginx can read `agents.conf` -/// without the rest of `/var/lib/hyperhive/` (forge/matrix tokens, etc.) -/// being exposed to the gateway container. -// nix: bind-mounted into the gateway container (hive-gateway.nix) — must match. +/// nginx runs on the host and reads this path directly; it used to be +/// bind-mounted into a gateway container at `/run/hive-state/`, exposing +/// this subdir *only* so the rest of `/var/lib/hyperhive/` (forge/matrix +/// tokens, etc.) stayed out of reach. On the host that narrowing is the +/// unit's sandbox, not a mount — nginx is not confined by this path. +// nix: named by the gateway's nginx config (hive-gateway/vhosts.nix) — must match. #[must_use] pub fn gateway_dir() -> PathBuf { state_root().join("gateway") diff --git a/hive-c0re/src/priv_client.rs b/hive-c0re/src/priv_client.rs index f2f45ed2..07a0f952 100644 --- a/hive-c0re/src/priv_client.rs +++ b/hive-c0re/src/priv_client.rs @@ -477,20 +477,21 @@ pub async fn register_ci_runner(token: &str) -> Result<()> { .await?) } -/// Restart a hive infrastructure container on the host (thin wrapper over -/// [`control_infra_container`] with `action = Restart`). hive-priv -/// re-validates `container` against its root-side allowlist; callers must -/// already have checked the requesting agent holds the `infra_admin` -/// capability. +/// Restart a hive infrastructure service on the host (thin wrapper over +/// [`control_infra_container`] with `action = Restart`). Callers must +/// already have checked that the requesting agent holds the `infra_admin` +/// capability *and* that the target is +/// [`agent_restartable`](InfraContainer::agent_restartable). pub async fn restart_infra_container(container: InfraContainer) -> Result<()> { control_infra_container(container, InfraAction::Restart).await } -/// Start / stop / restart a hive infrastructure container (`hive-ci`, +/// Start / stop / restart a hive infrastructure service (`hive-ci`, /// `hive-gateway`, `hive-forge`, `hive-matrix`) on the host via `systemctl -/// container@.service`. The [`InfraContainer`] enum is -/// the allowlist — hive-priv needs no name re-validation. Used by the -/// hive-wide `hivectl stop` / `hivectl start` flow. +/// `, where the unit is derived root-side from the variant +/// (`container@.service`, or `nginx.service` for the gateway). The +/// [`InfraContainer`] enum is the allowlist — hive-priv needs no name +/// re-validation. Used by the hive-wide `hivectl stop` / `start` flow. pub async fn control_infra_container(container: InfraContainer, action: InfraAction) -> Result<()> { ok(call(&PrivRequest::ControlInfraContainer { container, action }).await?) } diff --git a/hive-c0re/src/server.rs b/hive-c0re/src/server.rs index a68be5b8..7e6b9f63 100644 --- a/hive-c0re/src/server.rs +++ b/hive-c0re/src/server.rs @@ -863,7 +863,7 @@ async fn handle_stop( await_dags(coord, &queued, std::time::Duration::from_mins(2)).await; } for &container in infra { - let name = container.unit_name(); + let name = container.name(); match crate::priv_client::control_infra_container(container, InfraAction::Stop).await { Ok(()) => ok_items.push(name.to_owned()), Err(e) => { @@ -919,7 +919,7 @@ async fn handle_start( let mut errors: Vec = Vec::new(); for &container in infra { - let name = container.unit_name(); + let name = container.name(); match crate::priv_client::control_infra_container(container, InfraAction::Start).await { Ok(()) => ok_items.push(name.to_owned()), Err(e) => { @@ -990,7 +990,7 @@ async fn handle_restart_scoped( } for &container in &infra { - let name = container.unit_name(); + let name = container.name(); let res = async { crate::priv_client::control_infra_container(container, InfraAction::Stop).await?; crate::priv_client::control_infra_container(container, InfraAction::Start).await diff --git a/hive-c0re/src/socket_server/lifecycle_handlers.rs b/hive-c0re/src/socket_server/lifecycle_handlers.rs index 2f6fecd2..2567a41f 100644 --- a/hive-c0re/src/socket_server/lifecycle_handlers.rs +++ b/hive-c0re/src/socket_server/lifecycle_handlers.rs @@ -31,12 +31,13 @@ pub(super) async fn handle_start(coord: &Arc, agent: &str, name: &s /// orthogonal: it is gated on the `infra_admin` capability and audited, so it /// stays ahead of the topology guard. pub(super) async fn handle_restart(coord: &Arc, agent: &str, name: &str) -> Response { - // Infra-container restart: an agent holding the `infra_admin` - // capability can restart a hive infrastructure container (hive-ci / - // hive-gateway / hive-forge / hive-matrix) by passing its name to the - // same restart tool. The `InfraContainer` enum parse both recognises - // these (never agent children, so disjoint from the child path below) - // and yields the typed value the restart path needs. + // Infra restart: an agent holding the `infra_admin` capability can + // restart a hive infrastructure service (hive-ci / hive-forge / + // hive-matrix) by passing its name to the same restart tool. The + // `InfraContainer` enum parse both recognises these (never agent + // children, so disjoint from the child path below) and yields the typed + // value the restart path needs. It recognises `hive-gateway` too, which + // is then refused — a name the agent surface knows but may not act on. if let Ok(container) = name.parse::() { return handle_restart_infra(coord, agent, container).await; } @@ -60,7 +61,7 @@ async fn handle_restart_infra( agent: &str, container: hive_priv_sock::InfraContainer, ) -> Response { - let name = container.unit_name(); + let name = container.name(); // Record the attempt in the operator-visible privileged-action audit // trail, then emit a live `AuditEntryAdded` so the dashboard audit view // appends it off `/dashboard/stream`. Best-effort: `record` returns the @@ -75,6 +76,22 @@ async fn handle_restart_infra( coord.emit_audit_entry(entry); } }; + // Some targets are off-limits to agents regardless of capability — the + // gateway, because nginx fronts every hive service from the host and an + // agent bouncing it takes out the forge, the dashboard and matrix at + // once, including the route its own fix would have to travel. Checked + // before the capability so the refusal doesn't read as "ask for + // infra_admin"; no capability grants this. + if !container.agent_restartable() { + tracing::warn!(%agent, %name, "agent: infra restart denied (not agent-restartable)"); + audit( + crate::audit_log::AuditOutcome::Err, + Some("denied: target is not agent-restartable"), + ); + return Response::Err { + message: format!("`{name}` cannot be restarted by an agent; ask the operator"), + }; + } if !crate::capabilities::has_cap(agent, hive_sh4re::permissions::Capability::InfraAdmin) { tracing::warn!(%agent, %name, "agent: infra restart denied (no infra_admin capability)"); audit( diff --git a/hive-c0re/src/workers/agent_sockets.rs b/hive-c0re/src/workers/agent_sockets.rs index e00ef53b..b7a07b23 100644 --- a/hive-c0re/src/workers/agent_sockets.rs +++ b/hive-c0re/src/workers/agent_sockets.rs @@ -14,10 +14,11 @@ use std::path::{Path, PathBuf}; use anyhow::{Context, Result}; /// Host-side parent directory holding per-agent socket subdirs. The -/// gateway container bind-mounts this whole tree (read-only) so it -/// can `proxy_pass` to any agent. Each agent's container bind-mounts -/// only its own `/` subdir — agents can only access their own -/// sockets. The literal lives in `hive-host-sock` (shared with +/// gateway's nginx runs on the host and reads this whole tree, so it +/// can `proxy_pass` to any agent — it used to get there through a +/// read-only bind-mount of the same tree. Each agent's container +/// bind-mounts only its own `/` subdir, which is still what stops +/// one agent reaching another's socket. The literal lives in `hive-host-sock` (shared with /// `hivectl`); re-exported here under the name this module's consumers /// have always used. pub use hive_host_sock::AGENT_SOCKET_DIR; @@ -166,7 +167,7 @@ pub fn write(names: &[String]) -> Result<()> { /// /// Also calls `gateway_nginx::reload_if_pending` on every tick to /// retry a gateway nginx reload that may have failed on the previous -/// tick (e.g. gateway container temporarily down). This recovers +/// tick (e.g. gateway nginx temporarily down). This recovers /// gateway routing without needing a manual gateway restart. /// /// Mirrors the spawn-loop shape used by `crash_watch`, diff --git a/hive-host-sock/src/lib.rs b/hive-host-sock/src/lib.rs index ea993df3..7781d145 100644 --- a/hive-host-sock/src/lib.rs +++ b/hive-host-sock/src/lib.rs @@ -50,7 +50,7 @@ pub fn agent_state_dir(name: &Ident) -> PathBuf { /// `gateway/gateway.htpasswd` — nginx basic-auth credential store for the /// operator dashboard vhost. `hivectl`'s `--htpasswd-file` clap default. -// nix: read by the gateway container's nginx (hive-gateway.nix) — must match. +// nix: read by the gateway's nginx on the host (hive-gateway/) — must match. pub const GATEWAY_HTPASSWD: &str = "/var/lib/hyperhive/gateway/gateway.htpasswd"; /// `/run/hive-agent` — per-agent runtime socket dir root (web UI unix @@ -452,7 +452,8 @@ pub struct QuotaRow { /// (the bare `hivectl stop` / `start`); set individual fields to restrict /// (e.g. only `agents` → just the sub-agent containers). `agents` covers /// every managed sub-agent container; the rest are the named infra -/// containers (`hive-ci`, `hive-forge`, `hive-gateway`, `hive-matrix`). +/// services (`hive-ci`, `hive-forge`, `hive-gateway`, `hive-matrix`) — +/// containers, bar the gateway, which is the host's nginx unit. // // A flat bag of independent flag toggles — one bool per selectable // container class — is exactly the right shape here: each maps 1:1 to a diff --git a/hive-priv-sock/src/lib.rs b/hive-priv-sock/src/lib.rs index 1ec43637..3256c074 100644 --- a/hive-priv-sock/src/lib.rs +++ b/hive-priv-sock/src/lib.rs @@ -33,17 +33,24 @@ pub const MANAGER_NAME: &str = "ruth"; pub const AGENT_PREFIX: &str = "h-"; /// Sibling service containers managed by hive-c0re. This doubles as the -/// authoritative allowlist for infra lifecycle ops -/// ([`PrivRequest::ControlInfraContainer`]): any of these four may be -/// started / stopped / restarted (by the hive-wide `hivectl stop`/`start` -/// flow or an `infra_admin` agent's `restart`). `hive-c0re` is deliberately -/// absent — stopping it would sever the very socket the request arrived on. +/// authoritative allowlist for the requests that name a container as a +/// string (bind-mount edits, journal reads): only these — or a valid agent +/// name — are accepted. `hive-c0re` is deliberately absent; so is the +/// gateway, whose nginx is a plain host unit rather than a container. /// hive-priv re-validates against this list root-side, so it's authoritative /// regardless of what the caller sends. -pub const SIBLING_CONTAINERS: &[&str] = &["hive-forge", "hive-matrix", "hive-gateway", "hive-ci"]; +/// +/// ⚠️ This is the *container-name* allowlist, not the lifecycle one: +/// [`InfraContainer`] is what gates +/// [`PrivRequest::ControlInfraContainer`], and it has one variant more than +/// this list (the gateway). Keep the distinction — a name that belongs to +/// no container has no business reaching a `-M` / `nixos-container` call. +pub const SIBLING_CONTAINERS: &[&str] = &["hive-forge", "hive-matrix", "hive-ci"]; /// Lifecycle verb for [`PrivRequest::ControlInfraContainer`]. Maps directly -/// to `systemctl container@.service`. +/// to `systemctl `, where the unit comes from +/// [`InfraContainer::service_unit`] — usually `container@.service`, +/// but not always (see [`InfraTarget`]). #[derive(Debug, Clone, Copy, Serialize, Deserialize)] #[serde(rename_all = "snake_case")] pub enum InfraAction { @@ -71,8 +78,9 @@ impl InfraAction { /// by a runtime check. The c0re↔hive-priv wire form uses serde's default /// variant naming (`"Ci"`, `"Forge"`, …); it's an internal protocol (both /// ends rebuild together) so it needn't match the container name. -/// [`unit_name`](Self::unit_name) is the separate systemd / container name -/// (`hive-ci`). +/// [`name`](Self::name) is the separate stable identity string +/// (`hive-ci`), and [`service_unit`](Self::service_unit) the systemd unit +/// it actually resolves to. #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] pub enum InfraContainer { Ci, @@ -81,9 +89,27 @@ pub enum InfraContainer { Matrix, } +/// What an [`InfraContainer`] resolves to on the host — i.e. the thing a +/// lifecycle verb actually acts on. +/// +/// The gateway is why this exists: its nginx + dnsmasq were lifted out of +/// an nspawn container and onto the host, so "restart the gateway" is a +/// plain host unit now. The operator verb is unchanged; only its target +/// moved. Everything else is still a container. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum InfraTarget { + /// An nspawn container, controlled via `container@.service` and + /// readable with `journalctl -M `. + Container(&'static str), + /// A plain host unit. There is no machine to enter: no `-M` journal, + /// no `nixos-container` verb. + HostUnit(&'static str), +} + impl InfraContainer { - /// Every controllable infra container. The source of truth that - /// [`SIBLING_CONTAINERS`] is kept consistent with (see the test). + /// Every controllable infra target. A superset of + /// [`SIBLING_CONTAINERS`] — not all of these are containers (see the + /// test). pub const ALL: [InfraContainer; 4] = [ InfraContainer::Ci, InfraContainer::Forge, @@ -91,11 +117,14 @@ impl InfraContainer { InfraContainer::Matrix, ]; - /// The container / systemd-unit name, e.g. `hive-ci` → - /// `container@hive-ci.service`. (Distinct from the serde wire form, - /// which is the default variant name `"Ci"`.) + /// Stable identity string, e.g. `hive-ci`. This is what the operator + /// types, what the dashboard displays, and what [`FromStr`](std::str::FromStr) + /// parses — it stays `hive-gateway` even though the gateway is no + /// longer a container, because it names the *service*, not its + /// implementation. (Distinct from the serde wire form, which is the + /// default variant name `"Ci"`.) #[must_use] - pub fn unit_name(self) -> &'static str { + pub fn name(self) -> &'static str { match self { InfraContainer::Ci => "hive-ci", InfraContainer::Forge => "hive-forge", @@ -103,16 +132,48 @@ impl InfraContainer { InfraContainer::Matrix => "hive-matrix", } } + + /// Where this target lives on the host. + #[must_use] + pub fn target(self) -> InfraTarget { + match self { + InfraContainer::Gateway => InfraTarget::HostUnit("nginx.service"), + other => InfraTarget::Container(other.name()), + } + } + + /// The systemd unit a lifecycle verb acts on. + #[must_use] + pub fn service_unit(self) -> String { + match self.target() { + InfraTarget::Container(name) => format!("container@{name}.service"), + InfraTarget::HostUnit(unit) => unit.to_owned(), + } + } + + /// Whether an agent holding `infra_admin` may restart this target. + /// + /// The gateway is excluded by operator ruling: nginx now fronts every + /// hive service from the host, so an agent restarting it can take the + /// forge, dashboard and matrix down with it — including the path its + /// own PR would have to travel to fix it. The operator surface + /// (`hivectl`, dashboard) is unaffected. + #[must_use] + pub fn agent_restartable(self) -> bool { + !matches!(self, InfraContainer::Gateway) + } } impl std::str::FromStr for InfraContainer { type Err = (); - /// Parse a container name (`hive-ci`, …) into a variant. Used to decide - /// whether an MCP `restart()` target is a controllable infra - /// container. `Err(())` for anything that isn't one. + /// Parse an infra name (`hive-ci`, …) into a variant. Recognition + /// only — it says the name denotes a hive service, *not* that the + /// caller may act on it. The agent restart path additionally checks + /// [`agent_restartable`](InfraContainer::agent_restartable). + /// `Err(())` for anything that isn't one. fn from_str(s: &str) -> Result { - Self::ALL.into_iter().find(|c| c.unit_name() == s).ok_or(()) + Self::ALL.into_iter().find(|c| c.name() == s).ok_or(()) } } @@ -364,15 +425,25 @@ pub enum PrivRequest { /// Run `systemctl daemon-reload`. DaemonReload, - /// Synchronise the nginx unit inside the `hive-gateway` container. + /// Synchronise the host's nginx unit after an `agents.conf` write. /// /// hive-priv queries `ActiveState` and dispatches: /// - `active` → `systemctl reload nginx` (SIGHUP, zero-downtime) /// - `failed` → `systemctl reset-failed nginx` + `systemctl start nginx` /// - otherwise → `systemctl start nginx` /// - /// Requires root: `--machine=hive-gateway` enters the container - /// namespace via the machine bus (forbidden for unprivileged users). + /// Requires root because hive-c0re runs as the unprivileged + /// `hive-core` user and cannot act on a system unit. It used to be + /// root for a *different* reason — `--machine=hive-gateway` entering + /// the container's namespace over the machine bus — and that reason + /// died with the container: nginx is a host unit now. The + /// requirement survived the move; its justification did not. + /// + /// ⚠️ The unit name is **not** a parameter and must stay that way. + /// `--machine=` was doing double duty — transport *and* scope — so + /// dropping it removed the containment along with the namespace hop. + /// Hard-coding `nginx` is what replaces it: a caller cannot name the + /// unit, so this verb cannot be steered at any other service. ReloadGatewayNginx, // --- Forge admin CLI --- @@ -824,13 +895,14 @@ pub enum PrivEvent { #[cfg(test)] mod tests { - use super::{InfraContainer, SIBLING_CONTAINERS}; + use super::{InfraContainer, InfraTarget, SIBLING_CONTAINERS}; #[test] fn infra_control_allowlist_excludes_c0re_includes_matrix() { - // SIBLING_CONTAINERS is the authoritative allowlist for infra - // lifecycle ops. hive-c0re must NEVER be in it — stopping the daemon - // would sever the socket the request arrived on. + // SIBLING_CONTAINERS is the authoritative allowlist for the + // requests that name a container as a string. hive-c0re must NEVER + // be in it — stopping the daemon would sever the socket the request + // arrived on. assert!(!SIBLING_CONTAINERS.contains(&"hive-c0re")); // hive-matrix IS controllable (operator can stop/start/restart it). assert!(SIBLING_CONTAINERS.contains(&"hive-matrix")); @@ -838,26 +910,62 @@ mod tests { #[test] fn infra_container_enum_matches_sibling_containers() { - // The InfraContainer enum (the control-path allowlist) and the - // SIBLING_CONTAINERS slice (the general container-name validator) - // must list exactly the same four containers — they're separate - // surfaces for the same set, so keep them in lockstep. - let mut from_enum: Vec<&str> = InfraContainer::ALL.iter().map(|c| c.unit_name()).collect(); + // The two lists are the same set *minus the things that aren't + // containers*: every `Container` variant must appear in + // SIBLING_CONTAINERS and vice versa, so a `-M` journal read or a + // `nixos-container` verb can never be pointed at a name that has no + // machine behind it. + let mut from_enum: Vec<&str> = InfraContainer::ALL + .iter() + .filter_map(|c| match c.target() { + InfraTarget::Container(name) => Some(name), + InfraTarget::HostUnit(_) => None, + }) + .collect(); from_enum.sort_unstable(); let mut from_slice: Vec<&str> = SIBLING_CONTAINERS.to_vec(); from_slice.sort_unstable(); assert_eq!(from_enum, from_slice); + // The gateway is the one that is not: a host unit, and absent from + // the container-name allowlist. + assert_eq!( + InfraContainer::Gateway.target(), + InfraTarget::HostUnit("nginx.service") + ); + assert!(!SIBLING_CONTAINERS.contains(&"hive-gateway")); // hive-c0re has no variant — unrepresentable, can't be controlled. assert!("hive-c0re".parse::().is_err()); } #[test] fn infra_container_name_round_trips() { - // `unit_name` is the single source of truth for the wire form (the - // serde impls + FromStr all key off it), so a name→variant→name - // round-trip proves the mapping is consistent in both directions. + // `name` is the single source of truth for the operator-facing form + // (FromStr keys off it), so a name→variant→name round-trip proves + // the mapping is consistent in both directions. It holds for the + // gateway too: the name is still recognised, it's the *permission* + // that differs (see below), not the parse. for c in InfraContainer::ALL { - assert_eq!(c.unit_name().parse::(), Ok(c)); + assert_eq!(c.name().parse::(), Ok(c)); + } + } + + #[test] + fn service_unit_wraps_containers_but_not_host_units() { + assert_eq!( + InfraContainer::Ci.service_unit(), + "container@hive-ci.service" + ); + assert_eq!(InfraContainer::Gateway.service_unit(), "nginx.service"); + } + + #[test] + fn only_the_gateway_is_off_limits_to_agents() { + // Recognising a name and being allowed to restart it are separate + // questions — the gateway parses fine and is still refused. + assert!("hive-gateway".parse::().is_ok()); + assert!(!InfraContainer::Gateway.agent_restartable()); + for c in InfraContainer::ALL { + assert_eq!(c.agent_restartable(), c != InfraContainer::Gateway, "{c:?}"); } } } diff --git a/hive-priv/src/main.rs b/hive-priv/src/main.rs index 41b425fe..8d0f09b4 100644 --- a/hive-priv/src/main.rs +++ b/hive-priv/src/main.rs @@ -888,18 +888,23 @@ async fn register_ci_runner(token: &str) -> Result<(String, String)> { } /// `ControlInfraContainer` — start/stop/restart a hive infrastructure -/// container via `systemctl container@.service`. The -/// [`InfraContainer`] enum is the allowlist: serde already rejected any -/// unknown / unsafe name (hive-c0re has no variant, so a stop can't sever -/// the daemon socket) at deserialisation, so no root-side `.contains()` -/// check is needed here. Serves both the hive-wide `hivectl stop`/`start` -/// flow and an `infra_admin` agent's `restart` (action = Restart). +/// service via `systemctl `. The [`InfraContainer`] enum is +/// the allowlist: serde already rejected any unknown / unsafe name +/// (hive-c0re has no variant, so a stop can't sever the daemon socket) at +/// deserialisation, so no root-side `.contains()` check is needed here. +/// Serves both the hive-wide `hivectl stop`/`start` flow and an +/// `infra_admin` agent's `restart` (action = Restart). +/// +/// ⚠️ The unit is derived from the variant, never sent by the caller — +/// which is what keeps this from being a general `systemctl` pass-through. +/// It is not always `container@.service`: the gateway resolves to the +/// host's `nginx.service`. async fn control_infra_container( container: InfraContainer, action: InfraAction, ) -> Result<(String, String)> { let verb = action.systemctl_verb(); - let unit = format!("container@{}.service", container.unit_name()); + let unit = container.service_unit(); let out = Command::new("systemctl") .args([verb, &unit]) .output() @@ -2173,34 +2178,33 @@ async fn read_container_journal(container: &str, query: &JournalQuery) -> Result Ok((stdout, stderr)) } -/// Synchronise the nginx unit inside the `hive-gateway` container. +/// Synchronise the host's nginx unit after an `agents.conf` write. /// -/// Queries `ActiveState` via `systemctl --machine=hive-gateway` (requires -/// root — machine-bus transport enters the container namespace), then -/// dispatches: +/// Queries `ActiveState` and dispatches: /// - `active` → `systemctl reload nginx` (SIGHUP, zero-downtime) /// - `failed` → `systemctl reset-failed nginx` + `systemctl start nginx` /// - otherwise → `systemctl start nginx` /// +/// ⚠️ `nginx` is hard-coded on purpose — see `PrivRequest::ReloadGatewayNginx`. +/// The unit name is the scope of this verb, and it used to be enforced by +/// `--machine=hive-gateway` (which could only reach into that container). +/// With nginx on the host there is no namespace to bound it, so the +/// literal is the only thing standing between "reload the gateway" and +/// "reload anything". +/// /// Returns `(String::new(), String::new())` on success so it fits the /// `exec` return type directly. async fn sync_gateway_nginx() -> Result<(String, String)> { let state_out = Command::new("systemctl") - .args([ - "--machine=hive-gateway", - "show", - "--property=ActiveState", - "--value", - "nginx", - ]) + .args(["show", "--property=ActiveState", "--value", "nginx"]) .output() .await - .context("query nginx ActiveState in hive-gateway")?; + .context("query gateway nginx ActiveState")?; if !state_out.status.success() { tracing::warn!( exit_code = ?state_out.status.code(), stderr = %String::from_utf8_lossy(&state_out.stderr).trim(), - "systemctl show ActiveState exited non-zero — gateway container may be down" + "systemctl show ActiveState exited non-zero — gateway nginx may be down" ); } let state = String::from_utf8_lossy(&state_out.stdout).trim().to_owned(); @@ -2209,10 +2213,10 @@ async fn sync_gateway_nginx() -> Result<(String, String)> { match state.as_str() { "active" => { let out = Command::new("systemctl") - .args(["--machine=hive-gateway", "reload", "nginx"]) + .args(["reload", "nginx"]) .output() .await - .context("reload nginx in hive-gateway")?; + .context("reload gateway nginx")?; if !out.status.success() { bail!( "gateway nginx reload failed ({}): {}", @@ -2224,14 +2228,14 @@ async fn sync_gateway_nginx() -> Result<(String, String)> { "failed" => { // Clear start-limit so the next start can proceed. let _ = Command::new("systemctl") - .args(["--machine=hive-gateway", "reset-failed", "nginx"]) + .args(["reset-failed", "nginx"]) .status() .await; let out = Command::new("systemctl") - .args(["--machine=hive-gateway", "start", "nginx"]) + .args(["start", "nginx"]) .output() .await - .context("start nginx after reset-failed in hive-gateway")?; + .context("start gateway nginx after reset-failed")?; if !out.status.success() { bail!( "gateway nginx start (after reset-failed) failed ({}): {}", @@ -2243,10 +2247,10 @@ async fn sync_gateway_nginx() -> Result<(String, String)> { _ => { // inactive, activating, deactivating, unknown — just start. let out = Command::new("systemctl") - .args(["--machine=hive-gateway", "start", "nginx"]) + .args(["start", "nginx"]) .output() .await - .context("start nginx in hive-gateway")?; + .context("start gateway nginx")?; if !out.status.success() { bail!( "gateway nginx start failed (state={state:?}) ({}): {}", diff --git a/hive-sh4re/src/permissions.rs b/hive-sh4re/src/permissions.rs index 61d67cd3..cde4906c 100644 --- a/hive-sh4re/src/permissions.rs +++ b/hive-sh4re/src/permissions.rs @@ -224,11 +224,17 @@ pub enum Capability { /// manager socket for swarm-wide scans. QueryAgentState, /// Agent can restart hive infrastructure containers (hive-ci, - /// hive-gateway, hive-forge) via the `restart` MCP tool. hive-c0re + /// hive-forge, hive-matrix) via the `restart` MCP tool. hive-c0re /// checks this capability before routing the restart through /// hive-priv; the concrete service allowlist lives root-side in /// hive-priv. Deliberately generic ("infra admin") so future /// privileged infra ops can hang off the same grant. + /// + /// ⚠️ The gateway is **not** in reach of this capability, by operator + /// ruling — it is the host's nginx and fronts the forge, dashboard and + /// matrix, so an agent restarting it can cut the path its own fix + /// travels. That refusal is a property of the target, not of the + /// grant: no capability re-opens it. InfraAdmin, } @@ -265,7 +271,7 @@ impl Capability { "query non-child agents' loose ends and reminder state via get_loose_ends" } Self::InfraAdmin => { - "restart hive infrastructure containers (hive-ci, hive-gateway, hive-forge) via the restart tool" + "restart hive infrastructure containers (hive-ci, hive-forge, hive-matrix; not the gateway) via the restart tool" } } } diff --git a/hivectl/src/cli.rs b/hivectl/src/cli.rs index 6a32d3ad..4b8078c3 100644 --- a/hivectl/src/cli.rs +++ b/hivectl/src/cli.rs @@ -225,7 +225,7 @@ pub struct ScopeArgs { /// The forge container (`hive-forge`). #[arg(long)] forge: bool, - /// The gateway container (`hive-gateway`). + /// The gateway (`hive-gateway`) — nginx on the host, not a container. #[arg(long)] gateway: bool, /// The matrix container (`hive-matrix`). diff --git a/nix/host-modules/hive-gateway/default.nix b/nix/host-modules/hive-gateway/default.nix index 1ebd33c0..f6858f27 100644 --- a/nix/host-modules/hive-gateway/default.nix +++ b/nix/host-modules/hive-gateway/default.nix @@ -1,7 +1,10 @@ # Single nginx in front of every hyperhive web surface — dashboard, # per-agent UIs (sub-path), forge + matrix (sub-domain), .well-known -# delegations — plus the hive-internal dnsmasq resolver, co-located in -# the same `hive-gateway` container (shared host netns, state-free). +# delegations — plus the hive-internal dnsmasq resolver. Both run on the +# HOST, next to hive-c0re. They used to live in a `hive-gateway` +# container that shared the host netns anyway, so the boundary bought no +# network isolation and cost a resolv.conf sync, a reload that had to +# cross the machine bus, and four bind mounts. # Full vhost map + discovery flow + design rationale in # `docs/gateway.md`. Layout: ./options.nix (option declarations), # ./vhosts.nix (the nginx virtual-host tree), ./error-pages.nix @@ -42,6 +45,43 @@ let # so the gateway always terminates TLS. # `cfg.useSelfSigned` (options.nix) is the derived single source of truth. useSelfSigned = cfg.useSelfSigned; + + # nginx's own state dir. Kept at the historical `/var/lib/hive-gateway` + # path rather than renamed with the move: it holds the imported leaf + # across reboots, and renaming it would strand every existing hive's + # certs for no gain. + tlsDir = "/var/lib/hive-gateway/tls"; + # TLS cert + key paths. + # - self-signed (default): the hive-CA-signed leaf, imported into the + # state dir by `hive-gateway-self-signed-cert` below. + # - tls.certDir set: the operator's own cert dir, read directly. + tlsCert = + if cfg.tls.certDir != null then "${cfg.tls.certDir}/${cfg.tls.certName}" else "${tlsDir}/cert.pem"; + tlsKey = + if cfg.tls.certDir != null then "${cfg.tls.certDir}/${cfg.tls.keyName}" else "${tlsDir}/key.pem"; + # The swarm-services pair, used only by the vhosts whose names this + # hive's CA cannot sign. Self-signed mode only: with an operator cert + # or ACME the operator owns every name and there is no second issuer. + svcCert = "${tlsDir}/swarm-services.pem"; + svcKey = "${tlsDir}/swarm-services-key.pem"; + + nginxTree = import ./vhosts.nix { + inherit + lib + cfg + forgeCfg + matrixCfg + hyperhiveDomain + dashboardDist + swaggerUiTheme + tlsCert + tlsKey + svcCert + svcKey + swarmServiceDomains + ; + errorPages = import ./error-pages.nix { inherit pkgs; }; + }; in { imports = [ ./options.nix ]; @@ -65,10 +105,12 @@ in } ]; - # Ensure bind-mount sources exist at host boot before the gateway - # container's first start. nspawn would auto-create missing dirs; - # tmpfiles rules make the intent explicit and cover the fresh-boot - # window before c0re has run. + # Ensure the gateway state dirs exist at host boot, before anything + # reads or writes them. They used to double as bind-mount sources + # for the container (nspawn would auto-create a missing one); the + # rules stay because they still cover the fresh-boot window before + # c0re has run, and they pin owner + mode rather than leaving it to + # whoever creates the path first. # # /run/hive-agent — per-agent UDS socket dir, written by c0re's # set_nspawn_flags when agents start. Owned by `hive-core` (the @@ -95,349 +137,170 @@ in "f /var/lib/hyperhive/gateway/gateway.htpasswd 0644 root root - -" ]; - # Keep the gateway's `/etc/resolv.conf` in step with the host's. + # ⚠️ REMOVED WITH THE CONTAINER, and each one was a workaround for the + # boundary rather than a thing nginx or dnsmasq needed: # - # nixos-container does `cp --remove-destination /etc/resolv.conf - # "$root/etc/resolv.conf"` in its start script, and nspawn's - # `--resolv-conf=auto` copies rather than binds for a writable, - # host-netns container like this one. Both are one-shot: systemd-nspawn(1) - # says outright that "no further propagation of configuration is - # generally done after the one-time early initialization (this is - # because the file is usually updated through copying and renaming)". + # - `privateNetwork = false` — the container already shared the host + # netns, which is why nginx bound host ports and `localhost` + # upstreams reached hive-c0re. On the host that is simply true. + # - `additionalCapabilities = [ "CAP_NET_ADMIN" ]` — dnsmasq refuses + # to start with a `dhcp-range` unless it holds NET_ADMIN, and + # nspawn's bounding set dropped it for a host-netns container. + # Host root has it. + # - `networking.firewall.enable = false` — a container sharing the + # host netns would run *its* firewall.service against the HOST + # ruleset, flushing nixos-fw and deleting the nixos-nat-* chains on + # every boot. With one machine there is one firewall (below). + # - `networking.resolvconf.enable = false` + the `hive-gateway-resolv` + # path/service pair — the container's /etc/resolv.conf was a + # one-shot copy frozen at start, so a host network change left + # dnsmasq forwarding to a resolver that was gone. The whole + # watch-copy-reload machine existed to bridge two files. There is + # now one. + # - three bind mounts — /run/hive-agent, /run/hive-state, and either + # /run/hive-tls (operator cert) or /run/hive-ca (self-signed); + # those last two are mutually exclusive mkIfs, so it was never + # four. All plain host paths now. # - # So the gateway's copy is frozen at container start. dnsmasq has no - # explicit upstream (see ./dnsmasq.nix) and follows that file, which - # means a host network change — new router, new DHCP lease, laptop - # moving between networks — leaves dnsmasq forwarding to a resolver - # that is gone, and every non-hive lookup from every agent hangs. The - # agents' own resolvers point at the static bridge IP and never go - # stale, which is exactly why the symptom presents as "the gateway - # needs a kick". - # - # Hence: watch on the host, push into the container, reload dnsmasq. - # `reload` is `kill -HUP $MAINPID` (the upstream dnsmasq unit's - # ExecReload), so dnsmasq re-reads its upstream list and drops its - # cache without severing anything — nginx never notices. Why this has - # to run host-side, and why a file bind-mount is worse than the copy: - # `docs/network.md::Resolver behaviour`. - systemd.paths.hive-gateway-resolv = { - description = "Watch the host's /etc/resolv.conf for the hive-gateway container"; - wantedBy = [ "multi-user.target" ]; - # Arm the watch before anything configures the network, so the very - # first DHCP-driven resolv.conf write of the boot is caught too — - # that's the "gateway came up while DHCP was still settling" case. - # Inert if nothing pulls network-pre.target into the transaction. - before = [ "network-pre.target" ]; - pathConfig = { - # PathChanged also watches the parent directory, so openresolv's - # atomic rename-over lands as IN_MOVED_TO on /etc and triggers — - # a watch on the inode alone would die with the replaced file. - PathChanged = "/etc/resolv.conf"; - Unit = "hive-gateway-resolv.service"; - }; + # See `docs/network.md::Resolver behaviour` for the resolver history. + # ACME (Let's Encrypt) integration. nginx vhosts set + # `enableACME = true` via the vhost builder; this provides the + # shared ACME config (acceptTerms + email). + security.acme = lib.mkIf cfg.tls.acme.enable { + acceptTerms = true; + defaults.email = cfg.tls.acme.email; }; - systemd.services.hive-gateway-resolv = { - description = "Sync the host's resolvers into hive-gateway and reload dnsmasq"; - # Also run once per gateway start, to catch a host resolver change - # that happened while the container was down. Deliberately NOT - # ordered after network-online.target: pulling that target in on - # every resolv.conf change risks blocking the sync behind a - # wait-online timeout on hosts where nothing else reaches it. The - # boot race is closed by arming the path unit early instead. - wantedBy = [ "container@hive-gateway.service" ]; - after = [ "container@hive-gateway.service" ]; - path = [ - pkgs.systemd - pkgs.coreutils - pkgs.gnugrep - ]; + # Import the hive-CA leaf into nginx's state dir before nginx starts. + # + # 🚨 THIS LOOKS LIKE A LEFTOVER OF THE CONTAINER AND IS NOT — do not + # "simplify" it into pointing nginx at the CA dir. It does TWO jobs: + # + # (1) It re-modes the leaf. `hive-tls-ca` writes the key 0600 + # root:root; nginx's pre-start `nginx -t` runs as the nginx + # *user*, so a 0600 key fails the config test with + # `BIO_new_file() … Permission denied` and blocks the unit — + # hence the 0640 root:nginx copy below. That is a file-mode fact, + # not a namespace one, and it did not go away with the container. + # (2) It guarantees that **every cert path the nginx config names + # exists** — which is what the swarm-services fallback at the + # bottom of the script is for. nginx refuses to load a config + # naming a missing cert file, so a leaf that never issues takes + # the whole gateway down rather than one vhost; that has already + # happened once and it took the forge, dashboard and matrix with + # it. Removing this unit re-creates it exactly. + # + # nginx + # `Requires=` this via `requiredBy`, so it refuses to start until + # the copy succeeds. ALWAYS runs (no ConditionPathExists) and is + # idempotent — necessary to reconcile broken state from prior + # failed boots (a 0700 dir from a stale UMask, a truncated copy + # from an interrupted oneshot, etc.). The leaf covers the bare + # hive domain plus `forge.`, `matrix.` and `*.${hyperhiveDomain}` + # so all sub-domains validate under the same cert + the hive CA. + # See `docs/gateway.md` ("Self-signed TLS"). + systemd.services.hive-gateway-self-signed-cert = lib.mkIf useSelfSigned { + description = "Import host-generated TLS leaf for hive-gateway"; + wantedBy = [ "multi-user.target" ]; + before = [ "nginx.service" ]; + requiredBy = [ "nginx.service" ]; serviceConfig = { Type = "oneshot"; - SyslogIdentifier = "hive-gateway-resolv"; + RemainAfterExit = true; + # Pin the journal identity (else it's the `script` store-path wrapper). + SyslogIdentifier = "hive-gateway-self-signed-cert"; }; + path = [ pkgs.coreutils ]; script = '' - set -euo pipefail - src=/etc/resolv.conf - # Marker lives on tmpfs, so the first sync after every host boot - # always goes through rather than trusting a stale comparison. - marker=/run/hive-gateway/resolv.synced + set -eu + mkdir -p ${tlsDir} + # 0755 on BOTH the cert dir and its parent so the nginx + # user can traverse the full path. The parent + # `/var/lib/hive-gateway` lands at 0700 by default (systemd + # StateDirectory / mkdir umask depending on which service + # created it first), which on its own blocks traversal. + # Re-applied every boot in case a prior run left a tighter + # mode behind. + chmod 0755 ${builtins.dirOf tlsDir} + chmod 0755 ${tlsDir} + # Copy the host leaf in. `install` writes atomically with the + # target mode; run as root (container root == host root, + # privateUsers=false) so the 0600 root:root host key is + # readable. Key ends up root:nginx 0640 so nginx-pre-start + # (which runs `nginx -t` as the nginx user, not root) can + # read it — a 0600 root:root key passes the master load but + # fails the pre-start config test with `BIO_new_file() … + # Permission denied`, blocking the unit. Cert is world-read. + install -m 0644 ${config.services.hyperhive.tls.stateDir}/gateway.pem ${tlsCert} + install -m 0640 -g nginx ${config.services.hyperhive.tls.stateDir}/gateway-key.pem ${tlsKey} - # No nameserver line means either a mid-rewrite snapshot or a host - # with no DNS at all. In both cases the gateway's existing copy is - # the best information available — keep it and wait for the next - # event, instead of pushing a file that resolves nothing. - if ! grep -q '^[[:space:]]*nameserver[[:space:]]' "$src"; then - echo "host resolv.conf has no nameserver line — keeping the gateway's current copy" - exit 0 + # The swarm-services leaf, when this host issues one. It is + # a separate pair rather than more SANs on the one above + # because no hive CA can sign these names — each is + # constrained to its own hive's domain and the service + # names are siblings of it. + # + # Absent is a normal state, not a failure: the leaf exists + # only where the swarm CA is autoconfigured, and issuance + # can also fail on a host that wants one. + # + # ⚠️ When it is absent the HIVE leaf goes to this path + # anyway, and that fallback is load-bearing rather than + # tidy. nginx refuses to load a config naming a cert file + # that does not exist — `cannot load certificate … no such + # file` fails the pre-start test, so the vhost does not + # degrade, the ENTIRE gateway dies and takes the forge, the + # dashboard and matrix with it. Serving the hive leaf on a + # swarm-service name is a name mismatch: browsers warn, + # strict clients refuse, everything else keeps working, and + # the operator gets a bad cert instead of no hive. + # + # Measured, not theorised: this exact path took pr1ma's + # gateway down when the services sub-CA failed to issue. + if [ -s ${config.services.hyperhive.tls.stateDir}/swarm-services.pem ]; then + install -m 0644 ${config.services.hyperhive.tls.stateDir}/swarm-services.pem ${svcCert} + install -m 0640 -g nginx ${config.services.hyperhive.tls.stateDir}/swarm-services-key.pem ${svcKey} + else + echo "no swarm-services leaf — serving the hive leaf on those names (mismatch, not an outage)" >&2 + install -m 0644 ${config.services.hyperhive.tls.stateDir}/gateway.pem ${svcCert} + install -m 0640 -g nginx ${config.services.hyperhive.tls.stateDir}/gateway-key.pem ${svcKey} fi - - if [ -e "$marker" ] && cmp -s "$src" "$marker"; then - echo "host resolvers unchanged since last sync — nothing to do" - exit 0 - fi - - # A stopped gateway needs no push: its next start copies the - # current host file itself. - if ! systemctl is-active --quiet container@hive-gateway.service; then - echo "hive-gateway not running — its next start copies the current file itself" - exit 0 - fi - - # `machinectl copy-to` writes through the container's own mount - # namespace, so this stays correct regardless of how the container - # assembles /etc (e.g. if system.etc.overlay is ever turned on) — - # unlike poking at the rootfs path from the host side. - machinectl copy-to hive-gateway "$src" /etc/resolv.conf --force - - # Not fatal: dnsmasq polls resolv.conf for mtime changes on its own, - # so a failed reload costs a second of staleness, not correctness. - if ! systemctl -M hive-gateway reload dnsmasq.service; then - echo "dnsmasq reload failed (not up yet?) — copy is in place, its own poll will pick it up" - fi - - mkdir -p "$(dirname "$marker")" - install -m 0644 "$src" "$marker" - echo "synced host resolvers into hive-gateway and reloaded dnsmasq" ''; }; - containers.hive-gateway = { - autoStart = true; - ephemeral = false; - # Share host netns — nginx then binds host-level ports directly, - # `localhost` upstream resolution reaches hive-c0re without any - # port-forward dance, and the firewall config below is the only - # layer that matters. - privateNetwork = false; - # dnsmasq refuses to start once a dhcp-range is configured unless it - # holds CAP_NET_ADMIN (DNS-only mode doesn't need it). Private-network - # containers retain NET_ADMIN implicitly, but this container shares the - # host netns (above), so nspawn's default bounding set drops it — grant - # it explicitly. Note this is NET_ADMIN over the *host* netns; the - # gateway container is trusted infra (it already terminates TLS and - # fronts every vhost), so no new trust boundary is crossed. - additionalCapabilities = [ "CAP_NET_ADMIN" ]; - # Bind-mount the per-agent socket dir so nginx inside the gateway - # container can `connect(2)` to the UDS upstreams. - # Read-only (we just connect; harness writes the socket inside - # the agent's own container). Host-side dir is pre-created by a - # tmpfiles rule so nspawn always finds a source at boot. - bindMounts."/run/hive-agent" = { - hostPath = "/run/hive-agent"; - isReadOnly = true; - }; - # Bind-mount ONLY the gateway-specific subdir of the hyperhive - # state dir. Scoped to /var/lib/hyperhive/gateway/ rather than - # the whole parent so the gateway container can't read forge - # tokens or other files that may live at the parent level. - # c0re writes agents.conf under this subdir and triggers an nginx - # reload from the host via systemd-run after each write. - # Pre-created by a tmpfiles rule. - bindMounts."/run/hive-state" = { - hostPath = "/var/lib/hyperhive/gateway"; - isReadOnly = true; - }; - # Operator-provided TLS cert dir (e.g. Let's Encrypt / ACME). - # Only mounted when `tls.certDir` is set; when it is, the self-signed - # floor is off (so the `/run/hive-ca` mount below is absent). nginx - # reads cert + key from `/run/hive-tls/` and ``. - bindMounts."/run/hive-tls" = lib.mkIf (cfg.tls.certDir != null) { - hostPath = cfg.tls.certDir; - isReadOnly = true; - }; - # Self-signed mode: the host `hive-tls-ca` service generates a hive - # CA + a leaf signed by it under `services.hyperhive.tls.stateDir`. - # Bind-mount that dir read-only so the in-container import service - # (below) can copy the leaf into nginx's state dir with the right - # owner/mode. Source files: `gateway.pem` + `gateway-key.pem`. - bindMounts."/run/hive-ca" = lib.mkIf useSelfSigned { - hostPath = config.services.hyperhive.tls.stateDir; - isReadOnly = true; - }; - config = - { pkgs, ... }: - let - tlsDir = "/var/lib/hive-gateway/tls"; - # TLS cert + key paths inside the container. - # - self-signed (default): imported hive-CA-signed leaf in the - # persistent state dir. - # - tls.certDir set: operator-provided cert bind-mounted at /run/hive-tls. - tlsCert = - if cfg.tls.certDir != null then "/run/hive-tls/${cfg.tls.certName}" else "${tlsDir}/cert.pem"; - tlsKey = - if cfg.tls.certDir != null then "/run/hive-tls/${cfg.tls.keyName}" else "${tlsDir}/key.pem"; - # The swarm-services pair, used only by the vhosts whose names - # this hive's CA cannot sign. Self-signed mode only: with an - # operator cert or ACME the operator owns every name and there - # is no second issuer in the picture. - svcCert = "${tlsDir}/swarm-services.pem"; - svcKey = "${tlsDir}/swarm-services-key.pem"; - nginxTree = import ./vhosts.nix { - inherit - lib - cfg - forgeCfg - matrixCfg - hyperhiveDomain - dashboardDist - swaggerUiTheme - tlsCert - tlsKey - svcCert - svcKey - swarmServiceDomains - ; - errorPages = import ./error-pages.nix { inherit pkgs; }; - }; - in - { - system.stateVersion = "26.05"; + # nginx reload is triggered from the HOST side by hive-c0re + # after each agents.conf write, through hive-priv (c0re is + # unprivileged and cannot act on a system unit). + # + # It stays an explicit trigger rather than a systemd path unit + # watching the file. That used to be impossible — an IN_MOVED_TO + # from the atomic rename did not cross the nspawn mount-namespace + # boundary — and with one machine it would now work. It is still + # not wanted: the write and the reload belong in one causal chain + # c0re can retry and report on (see RELOAD_PENDING), not two + # independent units racing on an inotify event. - # This container shares the host netns, so its own - # firewall.service would run against the HOST ruleset: flush - # the nixos-fw chains, rebuild them from this container's - # (empty) port list, and delete the host's nixos-nat-* chains - # — wiping the bridge DHCP/DNS holes and the agents' NAT on - # every container boot. The host firewall owns all filtering; - # never run one in here. - networking.firewall.enable = false; + services.nginx = { + enable = true; + recommendedProxySettings = true; + recommendedTlsSettings = true; + recommendedGzipSettings = true; + recommendedOptimisation = true; + inherit (nginxTree) appendHttpConfig virtualHosts; + }; - # Keep the host-copied /etc/resolv.conf intact. nixos-container - # copies the host's file in at every container start, but - # resolvconf's host-tracking mode then regenerates it — to an - # empty file, since the host file doesn't cross the boundary - # after start (the same failure the matrix container hit). - # With resolvconf off, nothing in here touches the copy: nginx's - # own lookups (ACME) and dnsmasq's follow-the-host upstream - # default (see ./dnsmasq.nix) both read the host's resolvers. - # Keeping it stale-free is the host's job — see the - # `hive-gateway-resolv` path unit above. - networking.resolvconf.enable = false; - - # ACME (Let's Encrypt) integration. nginx vhosts set - # `enableACME = true` via the vhost builder; this provides the - # shared ACME config (acceptTerms + email). The gateway - # container has shared host netns so outbound ACME requests - # work without extra routing config. Certs are stored in the - # container's persistent state (`ephemeral = false`). - security.acme = lib.mkIf cfg.tls.acme.enable { - acceptTerms = true; - defaults.email = cfg.tls.acme.email; - }; - - # Import the host-generated leaf cert before nginx starts. - # The hive CA + gateway leaf are generated on the HOST by - # `hive-tls-ca` (see `hive-tls.nix`) and bind-mounted read-only - # at `/run/hive-ca`; this service copies the leaf into nginx's - # state dir with the owner/mode nginx needs, rather than reading - # the bind-mount directly (the host key is 0600 root:root and a - # cross-namespace bind-mount can't be relaxed in place). nginx - # `Requires=` this via `requiredBy`, so it refuses to start until - # the copy succeeds. ALWAYS runs (no ConditionPathExists) and is - # idempotent — necessary to reconcile broken state from prior - # failed boots (a 0700 dir from a stale UMask, a truncated copy - # from an interrupted oneshot, etc.). The leaf covers the bare - # hive domain plus `forge.`, `matrix.` and `*.${hyperhiveDomain}` - # so all sub-domains validate under the same cert + the hive CA. - # See `docs/gateway.md` ("Self-signed TLS"). - systemd.services.hive-gateway-self-signed-cert = lib.mkIf useSelfSigned { - description = "Import host-generated TLS leaf for hive-gateway"; - wantedBy = [ "multi-user.target" ]; - before = [ "nginx.service" ]; - requiredBy = [ "nginx.service" ]; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - # Pin the journal identity (else it's the `script` store-path wrapper). - SyslogIdentifier = "hive-gateway-self-signed-cert"; - }; - path = [ pkgs.coreutils ]; - script = '' - set -eu - mkdir -p ${tlsDir} - # 0755 on BOTH the cert dir and its parent so the nginx - # user can traverse the full path. The parent - # `/var/lib/hive-gateway` lands at 0700 by default (systemd - # StateDirectory / mkdir umask depending on which service - # created it first), which on its own blocks traversal. - # Re-applied every boot in case a prior run left a tighter - # mode behind. - chmod 0755 ${builtins.dirOf tlsDir} - chmod 0755 ${tlsDir} - # Copy the host leaf in. `install` writes atomically with the - # target mode; run as root (container root == host root, - # privateUsers=false) so the 0600 root:root host key is - # readable. Key ends up root:nginx 0640 so nginx-pre-start - # (which runs `nginx -t` as the nginx user, not root) can - # read it — a 0600 root:root key passes the master load but - # fails the pre-start config test with `BIO_new_file() … - # Permission denied`, blocking the unit. Cert is world-read. - install -m 0644 /run/hive-ca/gateway.pem ${tlsCert} - install -m 0640 -g nginx /run/hive-ca/gateway-key.pem ${tlsKey} - - # The swarm-services leaf, when this host issues one. It is - # a separate pair rather than more SANs on the one above - # because no hive CA can sign these names — each is - # constrained to its own hive's domain and the service - # names are siblings of it. - # - # Absent is a normal state, not a failure: the leaf exists - # only where the swarm CA is autoconfigured, and issuance - # can also fail on a host that wants one. - # - # ⚠️ When it is absent the HIVE leaf goes to this path - # anyway, and that fallback is load-bearing rather than - # tidy. nginx refuses to load a config naming a cert file - # that does not exist — `cannot load certificate … no such - # file` fails the pre-start test, so the vhost does not - # degrade, the ENTIRE gateway dies and takes the forge, the - # dashboard and matrix with it. Serving the hive leaf on a - # swarm-service name is a name mismatch: browsers warn, - # strict clients refuse, everything else keeps working, and - # the operator gets a bad cert instead of no hive. - # - # Measured, not theorised: this exact path took pr1ma's - # gateway down when the services sub-CA failed to issue. - if [ -s /run/hive-ca/swarm-services.pem ]; then - install -m 0644 /run/hive-ca/swarm-services.pem ${svcCert} - install -m 0640 -g nginx /run/hive-ca/swarm-services-key.pem ${svcKey} - else - echo "no swarm-services leaf — serving the hive leaf on those names (mismatch, not an outage)" >&2 - install -m 0644 /run/hive-ca/gateway.pem ${svcCert} - install -m 0640 -g nginx /run/hive-ca/gateway-key.pem ${svcKey} - fi - ''; - }; - - # nginx reload is triggered from the HOST side by hive-c0re - # via `systemctl -M hive-gateway reload nginx` after each - # agents.conf write — letting systemd resolve the nginx binary - # path avoids exit-203 EXEC failures. A path unit watching the - # bind-mounted file inside the container does not work: an - # IN_MOVED_TO from an atomic rename on the host does not - # propagate across the nspawn mount-namespace boundary. The - # host-side trigger is the correct approach. - - services.nginx = { - enable = true; - recommendedProxySettings = true; - recommendedTlsSettings = true; - recommendedGzipSettings = true; - recommendedOptimisation = true; - inherit (nginxTree) appendHttpConfig virtualHosts; - }; - - services.dnsmasq = import ./dnsmasq.nix { - inherit - lib - networkCfg - forgeCfg - matrixCfg - hyperhiveDomain - ; - }; - }; + # dnsmasq moves with nginx rather than staying behind: it was only in + # the container because nginx was, and leaving it there would keep the + # whole resolv.conf sync machine alive for a resolver that no longer + # needs it. Host-side it reads the one /etc/resolv.conf directly. + services.dnsmasq = import ./dnsmasq.nix { + inherit + lib + networkCfg + forgeCfg + matrixCfg + hyperhiveDomain + ; }; networking.firewall = lib.mkIf cfg.openFirewall { diff --git a/nix/host-modules/hive-gateway/options.nix b/nix/host-modules/hive-gateway/options.nix index 41e30a77..772ab8d8 100644 --- a/nix/host-modules/hive-gateway/options.nix +++ b/nix/host-modules/hive-gateway/options.nix @@ -134,9 +134,9 @@ in and uses this cert, overriding the self-signed default — the auto-generated hive-CA-signed leaf is skipped entirely. - The directory is bind-mounted read-only into the gateway - container at `/run/hive-tls/`. nginx reads - `/` and `/`. + nginx reads `/` and + `/` directly — it runs on the host, so + the directory needs no bind mount and no copy. Default filenames (`cert.pem` / `key.pem`) match the output layout of nixpkgs's `security.acme` module. @@ -232,9 +232,7 @@ in enabled, every request to the gateway's main vhost requires a valid username and password. nginx's built-in `auth_basic` module validates credentials against - `/var/lib/hyperhive/gateway/gateway.htpasswd` on the host - (exposed as `/run/hive-state/gateway.htpasswd` inside the - container via the existing gateway state bind-mount). Off by default. + `/var/lib/hyperhive/gateway/gateway.htpasswd`. Off by default. Manage users with `hivectl gateway create-user`, `delete-user`, and `list-users` — see `hivectl gateway --help` for usage. diff --git a/nix/host-modules/hive-gateway/vhosts.nix b/nix/host-modules/hive-gateway/vhosts.nix index 8cf647d7..91d5f484 100644 --- a/nix/host-modules/hive-gateway/vhosts.nix +++ b/nix/host-modules/hive-gateway/vhosts.nix @@ -241,7 +241,7 @@ let # `/agent/` catch-all 404 + the two internal error-page targets it # points at. Per-agent `location /agent//` blocks live in the - # runtime-generated `/run/hive-state/agents.conf` (included via + # runtime-generated `/var/lib/hyperhive/gateway/agents.conf` (included via # `extraConfig` on the vhost); nginx longest-prefix-match makes a # real `/agent//` beat this catch-all. `internal` keeps the # error pages reachable only through nginx's error handling. @@ -275,7 +275,7 @@ let # secret (`X-Hub-Signature-256`) protects those endpoints instead. dashboardAuth = lib.optionalString cfg.auth.enable '' auth_basic "${cfg.auth.realm}"; - auth_basic_user_file /run/hive-state/gateway.htpasswd; + auth_basic_user_file /var/lib/hyperhive/gateway/gateway.htpasswd; # `=401` keeps the status 401 so the login dialog shows; the # internal page explains `hivectl gateway create-user`. error_page 401 =401 /__hive_auth_unauthorized; @@ -397,15 +397,14 @@ in }; # Per-agent location blocks, generated at runtime by # hive-c0re and written to /var/lib/hyperhive/gateway/agents.conf - # on the host. The bind-mount at /run/hive-state/ exposes - # that file here. nginx parses `include` at config-load - # time so a reload (triggered by c0re via systemd-run + # on the host — the same machine nginx runs on. nginx parses + # `include` at config-load time so a reload (triggered by c0re # after each agents.conf write) picks up new or removed # agents without a nixos-rebuild. nginx's longest-prefix- # match rule ensures `/agent//` from this file beats # the `/agent/` catch-all above. extraConfig = securityHeaders + '' - include /run/hive-state/agents.conf; + include /var/lib/hyperhive/gateway/agents.conf; ''; }; } diff --git a/swarm-controller/README.md b/swarm-controller/README.md index 0d896e85..2cff09ef 100644 --- a/swarm-controller/README.md +++ b/swarm-controller/README.md @@ -35,17 +35,18 @@ The socket is `0666`. It has to be: nginx runs as a different user and sockets, and rests on the same argument — *"the bind source dir is per-agent on host so blast radius is unchanged."* -What keeps that safe is that the directory holds **one** socket and is -bind-mounted into **one** container. So: +What keeps that safe is that the directory holds **one** socket. So: > **Never point `socketPath` at a directory that carries anything else.** > `/run/hyperhive` above all — it holds `host.sock`, the host **admin** socket. -> Mounting that directory to reach this socket would hand the gateway container -> the admin socket along with it. +> Pointing nginx at that directory to reach this socket would put the admin +> socket within its reach too. -Changing `socketPath` therefore means re-checking the gateway bind-mount, not -just the daemon. A unit test pins the default path so a tidying edit fails -instead of reviewing cleanly. +This got *less* forgiving when nginx moved onto the host: the gateway used to +reach a unix upstream through a bind-mount, so the mount list was a second +bound on what it could touch. There is no mount now — the directory is the +whole of the access control. A unit test pins the default path so a tidying +edit fails instead of reviewing cleanly. `RuntimeDirectoryPreserve=yes` and the daemon's stale-socket unlink on start are a **pair**: preserving the directory without the unlink means `bind` fails with diff --git a/swarm-controller/src/main.rs b/swarm-controller/src/main.rs index aefb3a75..58a16727 100644 --- a/swarm-controller/src/main.rs +++ b/swarm-controller/src/main.rs @@ -27,10 +27,12 @@ use axum::{Router, routing::get}; /// `ExecStart`, so the default names a directory the unit just produced. /// /// The directory is its own — deliberately not shared with hive-c0re's -/// `/run/hyperhive`. nginx reaches a unix upstream by having the socket's -/// *directory* bind-mounted into the gateway container, so co-locating -/// this socket with c0re's admin socket would hand the gateway the admin -/// socket along with it. +/// `/run/hyperhive`. The socket is `0666`, so its directory is the only +/// access control it has; co-locating it with c0re's admin socket would +/// put both within reach of whatever can reach either. That used to be +/// enforced by which *directory* was bind-mounted into the gateway +/// container; with nginx on the host the mount is gone and the directory +/// is all that is left, so the rule matters more, not less. const DEFAULT_SOCKET: &str = "/run/swarm-controller/controller.sock"; fn socket_path() -> PathBuf { @@ -96,8 +98,9 @@ mod tests { /// The socket must not share a directory with anything else, because /// the socket is `0666` and the directory is therefore the only access /// control it has. `/run/hyperhive` in particular holds hive-c0re's - /// **admin** socket, and nginx reaches a unix upstream by mounting the - /// socket's whole directory into the gateway container. + /// **admin** socket. nginx used to reach a unix upstream by mounting + /// the socket's whole directory into the gateway container; it runs on + /// the host now, so nothing narrows its reach but the directory itself. /// /// A test rather than a comment: the failure this guards against is a /// one-word edit that looks tidier and reads fine in review. @@ -111,7 +114,7 @@ mod tests { Path::new("/run/swarm-controller"), "the socket's directory is its access control — moving it under a shared \ directory (notably /run/hyperhive, which holds the host admin socket) \ - exposes everything else in that directory to the gateway container" + exposes everything else in that directory to the gateway's nginx" ); } }