`agents.conf` and `gateway.htpasswd` move from /var/lib/hyperhive/gateway
to /var/lib/hive-gateway/conf, alongside the `tls/` the gateway already
kept there.
nginx reads both as an unprivileged user. Under c0re's state dir it could
only reach them by traversing a directory systemd re-declares `0750
hive-core` on every c0re start — so nginx was given `SupplementaryGroups
= [ "hive-core" ]`, which also handed it read access to everything else
group-readable in that tree. The tokens are individually 0600, but the
broker sqlite carries no explicit mode: every message between every agent
was readable by the process whose job is parsing untrusted network input.
Moving the files removes the need and the exposure together. The group is
gone, and its absence is now commented as load-bearing so it doesn't come
back as a fix for a symptom it would recreate.
Also drops this module's `/var/lib/hyperhive` tmpfiles rule. It declared
`0755 root root` and could never win against `StateDirectoryMode`, and a
losing declaration still reads as a guarantee — that is what sent the
first diagnosis of the outage looking for who had changed the mode.
Ordering is unchanged and still the thing that makes a fresh boot work:
tmpfiles runs before services and seeds both files empty-but-valid, nginx
names them (an `include` of a missing file is fatal, not empty), and
content arrives when c0re writes and reloads — which it does on every
topology change, so a boot against the empty seed resolves itself.
Folds in the mode fix: `write` now sets 0644 on the tmp file before the
rename, because a rename carries the source's mode and discards the
destination's, and the tmpfiles rule that declares 0644 is
create-if-absent so it never re-applies.
Per review: docs represent current state. Every "used to" / "no longer"
clause this branch introduced is gone — including the History section in
network.md, which was a whole subsection about a sync mechanism that
doesn't exist.
Where the removed clause was carrying a real constraint, the constraint
stays and is stated in the present tense instead of as a delta: nothing
narrows what the gateway's nginx can reach except the directory
permissions in front of a socket, and nothing bounds `ReloadGatewayNginx`
except the hard-coded unit name. Those read as rules now rather than as
the story of how they came to be rules.
The gateway's nginx + dnsmasq no longer run in their own nspawn container.
`nix/host-modules/hive-gateway/default.nix` loses the
`containers.hive-gateway` wrapper and everything that existed only to punch
holes in it: `privateNetwork = false`, `CAP_NET_ADMIN`, five bind mounts,
its own `stateVersion`, `networking.firewall.enable = false`,
`networking.resolvconf.enable = false`, and the `hive-gateway-resolv`
path+service pair. 465 -> 303 lines.
The container never bought isolation here. It shared the host netns by
necessity — nginx binds the host's :80/:443, dnsmasq answers on the bridge —
so each of those settings was undoing a boundary the gateway could not
afford in the first place.
Four things made it more than a deletion, none of them visible in the nix
diff:
- The self-signed cert service also imports the hive CA leaf, so removing it
with the container would have left nginx naming a missing cert file, which
it refuses to load at all.
- The nginx reload is a hive-priv verb. It still needs root, but no longer
for the reason its doc gave, and `--machine=` was both transport and
scope — so the unit name is now hard-coded in the helper as the
containment.
- The lifecycle verb named a container that stops existing.
- `journalctl -M hive-gateway` had no machine to enter.
Per the operator's ruling, the operator verb keeps working and agents lose
it. `InfraContainer` answered three questions that used to share an answer;
it now splits into `name()` (identity), `target()` (Container vs HostUnit),
`service_unit()` (the systemd unit), and `agent_restartable()`, which the
MCP restart path checks before the capability so the refusal cannot read as
"ask for infra_admin". `SIBLING_CONTAINERS` drops the gateway — it gates the
requests that name a container as a string — while `FromStr` still accepts
it, because that answers what a name is, not who may act on it. The
dashboard's gateway journal reads host journald filtered to `nginx.service`.
Prose was corrected where it only named a location, and re-argued where the
container was doing security work: a `0666` per-agent socket was safe
because only the gateway container had the directory bind-mounted. There is
no mount now, so the directory permissions are the whole of the access
control — the constraint holds, its mechanism doesn't.
Gate: nix fmt / clippy --all-targets -D warnings / cargo test all clean (710
tests); hivectl-cli.md regenerated from the clap tree. The nix eval was run
in both TLS shapes at this commit: every delta in the rendered
virtualHosts is one of the three intended path moves, dnsmasq settings are
byte-identical, and the absence probe flips true -> false with bindMounts
emptied.
Extends the SweepHealth/warnings registry (already landed for
knowledge_pull) to two more background sweeps:
- gateway_nginx::reload_gateway_nginx: raises a warn-level banner
immediately on the first failed reload (routing changes silently
not taking effect is user-visible right now, so no debounce).
- forge::repos::ensure_config_repo: raises a crit-level banner
listing every agent whose config-repo branch protection is
currently unapplied (security-relevant — bypasses the deploy
pipeline), clearing agents out of the message as they recover.
Journal warn!/error! logging is left in place; the registry adds a
dashboard-visible signal on top. forge::ensure_all() and
matrix::ensure_all() sweeps are deliberately left for a fast-follow.
Agents run in private netns (always-on isolation). The TCP fallback
to 127.0.0.1:<port> was unreachable from the gateway's host netns
regardless of whether the per-agent socket marker existed.
Remove the conditional entirely: always use the unix socket path.
If the socket is not yet bound, nginx returns 502 which is already
handled by the error_page 502 503 504 = /__hive_agent_unreachable
directive in every location block.
Also removes the unused lifecycle::agent_web_port call and the
now-misleading '.bound state' mention from the render doc comment.
systemctl --machine=hive-gateway requires root (machine-bus transport
enters the container namespace). hive-c0re is unprivileged, so every
call to nginx_active_state() and gateway_systemctl() silently failed
with exit 1, causing a continuous 30s retry loop without ever
syncing nginx.
Fix:
- Move state-aware nginx logic into hive-priv ReloadGatewayNginx:
check ActiveState, then reload/reset-start/start accordingly.
hive-priv already runs as root and has machine-bus rights.
- Remove nginx_active_state() and gateway_systemctl() from
gateway_nginx.rs (they were always running unprivileged, always
failing silently).
- Make write(), reload_if_pending(), reload_gateway_nginx() async so
they can call the async priv_client without a blocking bridge.
- Update callers in agent_sockets::spawn_poll and meta::sync_agents
to await the now-async functions.
The priv_client::reload_gateway_nginx() call and PrivRequest::ReloadGatewayNginx
wire type already existed — the gateway_nginx module was just not using them.
RELOAD_PENDING stays true on any failed reload, and reload_if_pending
fires on every 10-second spawn_poll tick. When the gateway is down or
nginx config is bad this hammers systemctl indefinitely.
Fix: track LAST_FAILED_RELOAD (unix timestamp). reload_if_pending backs
off to one attempt per RELOAD_RETRY_SECS (30s) after a failure. Fresh
write() calls reset the backoff so new topology changes still attempt
reload immediately.
Fixes#1113.
All crates now pass `cargo clippy --workspace -- -D warnings` cleanly.
Fixes span six crates (hive-sh4re, hive-ag3nt, hive-c0re, hive-forge,
hive-priv, hive-matrix-mcp was already clean):
- doc_markdown: wrap snake_case, type names, constants in backticks
- collapsible_if / collapsible_match: fold nested ifs into let-chains
- duration_suboptimal_units: Duration::from_secs(N) → from_mins/from_hours
- implicit_hasher: allow on HashMap-param fns where generalization is risky
- items_after_statements: hoist use to function tops
- map(f).unwrap_or(x) → map_or(x, f); map(f).unwrap_or_else(g) → map_or_else
- is_ok_and / is_none_or in place of map().unwrap_or(bool)
- needless_continue: {} instead of continue in loop match arms
- match_same_arms: Ok(None) | Err(_) merged
- format_push_str: write!() instead of push_str(&format!())
- while let replaces loop { let Some(..) = x else { break } }
- struct_excessive_bools / dead_code: allow on purpose-built structs
- too_many_lines / too_many_arguments: allow where refactor not worth it
- unused_async: remove async from poll_once in bash_runner
- needless_borrow: fix &repo deref in hive-forge comments verb
- cast_possible_truncation: allow u64→usize in fetch_tail
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nginx -s reload signals a running master process. When nginx enters
failed state (start-limit-hit from repeated nginx -t failures on a
bad agents.conf), there is no master and the reload is a silent no-op.
c0re kept re-firing the same no-op reload forever via RELOAD_PENDING.
Fix: probe the nginx unit's ActiveState before sending the reload:
- active → nginx -s reload (existing zero-downtime path)
- failed → systemctl reset-failed nginx + systemctl start nginx
- other → systemctl start nginx
This makes c0re self-healing: once a corrected agents.conf is published,
the next reload_gateway_nginx call clears the start-limit and restarts
nginx automatically without operator intervention.
New helpers: nginx_active_state() (systemctl show --property=ActiveState
--value) and gateway_systemctl() (host-side systemctl --machine=hive-gateway).
nginx forbids a URI part on proxy_pass inside named locations, so the
split-mode `@<name>_dynamic` fallback (introduced when the gateway
started serving static dist directly) failed config-test with:
"proxy_pass" cannot have URI part in location given by regular
expression, or inside named location
Render two upstream forms: prefix locations keep the trailing slash so
nginx auto-strips the location prefix; the named-location path strips
`/agent/<name>/` via `rewrite ... break` and uses a bare upstream.
gateway_nginx.rs reads HIVE_AGENT_FRONTEND_DIR (injected by hive-c0re.nix
as ${cfg.frontend}/agent). When set, agents.conf emits per-agent split
blocks instead of the old single proxy_pass:
# Compiled assets — immutable nix store path, cache 1y
location ^~ /agent/<name>/static/ {
alias <frontend>/static/;
expires 1y; add_header Cache-Control "public, immutable, ...";
}
# Static dist + proxy fallback
location /agent/<name>/ {
alias <frontend>/;
try_files $uri $uri.html $uri/index.html @<name>_dynamic;
}
location @<name>_dynamic {
proxy_pass <upstream>; # api, events, icon, login, …
…proxy headers unchanged…
}
try_files path resolution (nginx applies alias mapping first):
$uri — exact file (/static/app.js → static/app.js)
$uri.html — bare-path fallback (/stats → stats.html)
$uri/index.html — directory index (/ → index.html)
@<name>_dynamic — proxy catchall for anything not in the dist
Adding pages to the frontend dist works automatically — no generator
change needed. Per-agent extraFiles (in mergedDist, not in the base
nix-store path) continue to proxy to the agent daemon.
frontend is a nix store path injected at build time — only [a-z0-9/._-],
no shell metacharacters — safe to interpolate without sanitization;
comment added documenting this assumption.
Without HIVE_AGENT_FRONTEND_DIR the existing single-proxy block is
emitted unchanged — backward-compatible for deployments without the env.
render() takes frontend_dir as a parameter so tests exercise both code
paths safely in parallel. 13 tests: 7 legacy, 6 split-mode. No clippy
warnings in changed files.
nix/modules/hive-c0re.nix: inject HIVE_AGENT_FRONTEND_DIR = "${cfg.frontend}/agent".
The #872 path-unit approach was silently broken: IN_MOVED_TO from an
atomic rename on the host does not propagate across the nspawn
mount-namespace boundary into the container, so the watcher inside
the gateway container never fired.
Fix: after each agents.conf write, c0re calls
`systemd-run --machine=hive-gateway -- nginx -s reload` from the host.
The reload is best-effort (logged on failure, not fatal).
Remove the now-unused `hive-gateway-agents-conf.path` +
`hive-gateway-nginx-reload.service` from the gateway container config.
Update docs/gateway.md + comments to reflect the host-side approach.
- gateway_nginx.rs: HOST_CONF_PATH → /var/lib/hyperhive/gateway/agents.conf
- hive-gateway.nix: hostPath = /var/lib/hyperhive/gateway (not whole parent
dir — avoids exposing forge tokens or other credentials to the gateway
container)
- tmpfiles: add /var/lib/hyperhive/gateway/ dir rule + seed agents.conf there
- scrub "(#869)" from hive-gateway-nginx-reload service description
Replace eval-time per-agent nginx location baking with a runtime
include file. c0re writes /var/lib/hyperhive/agents.conf (nginx
location blocks, UDS or TCP per agent) on every topology change and
on the 10s marker poll. The gateway container bind-mounts
/var/lib/hyperhive/ at /run/hive-state/ and nginx includes
/run/hive-state/agents.conf. A systemd path unit inside the container
watches the file for changes and fires `nginx -s reload` on each
atomic rename from c0re — no nixos-rebuild switch needed when agents
start, stop, or flip useUnixSocket.
- new hive-c0re/src/gateway_nginx.rs: write() + render()
- lib.rs + meta.rs + agent_sockets::spawn_poll: hook in write()
- hive-gateway.nix: drop agentPortsTable/agentSocketsTable/
agentUpstreamFor/lib.mapAttrs', add /run/hive-state bind-mount,
include directive, systemd path unit + reload service, tmpfiles
for /var/lib/hyperhive + agents.conf seed
- docs/gateway.md: update vhost table + Per-agent UDS section
