vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(#970,#973): retry gateway reload on failure; always enable HIVE_WEB_SOCKET

Browse source
This commit is contained in:
damocles 2026-06-01 18:18:45 +02:00
commit b83edc40c6
5 changed files with 88 additions and 82 deletions
Download patch file Download diff file Expand all files Collapse all files

33
docs/gateway.md
View file

@ -71,13 +71,14 @@ SSH for forge stays direct on `cfg.sshPort` — separate listener protocol, not
## Per-agent unix-socket upstream ## Per-agent unix-socket upstream
Sub-agent `/agent/<name>/` upstreams flip from TCP loopback to a All agents bind their web UI on a unix-domain socket at
unix-domain socket as each agent opts in. The mechanism: `/run/hive-agent/<name>/web.sock` — the `HIVE_WEB_SOCKET` env var is
now set unconditionally for every agent. The mechanism:
1. **Agent side** (`hyperhive.web.useUnixSocket = true` in 1. **Agent side**. `HIVE_WEB_SOCKET=/run/hive-agent/<name>/web.sock`
`agent.nix`). Sets `HIVE_WEB_SOCKET=/run/hive-agent/<name>/web.sock` is set on every harness service env; `web_ui::serve` binds a
on the harness service env; `web_ui::serve` binds a `UnixListener` `UnixListener` at that path. The deprecated
at that path instead of TCP. `hyperhive.web.useUnixSocket` option is now a no-op.
2. **Host side**. `hive-c0re` bind-mounts the per-agent subdir 2. **Host side**. `hive-c0re` bind-mounts the per-agent subdir
(`/run/hive-agent/<name>/`) into the agent's container. Dir (`/run/hive-agent/<name>/`) into the agent's container. Dir
bind, not file bind — file bind-mounts don't survive the bind, not file bind — file bind-mounts don't survive the
@ -87,32 +88,28 @@ unix-domain socket as each agent opts in. The mechanism:
`<dir>/hyperhive-socket-bound` next to the socket. c0re's `<dir>/hyperhive-socket-bound` next to the socket. c0re's
`agent_sockets::write` filters its JSON map by marker presence — `agent_sockets::write` filters its JSON map by marker presence —
only agents whose harness has actually bound the socket appear there. only agents whose harness has actually bound the socket appear there.
Without this filter, the gateway would `proxy_pass` to a non-existent (Legacy name `.bound` also accepted during the transition window.)
socket for every sub-agent that hasn't opted in yet. (Legacy name
`.bound` also accepted during the transition window.)
4. **Gateway side**. `gateway_nginx::write` generates 4. **Gateway side**. `gateway_nginx::write` generates
`/var/lib/hyperhive/gateway/agents.conf` — a plain nginx include `/var/lib/hyperhive/gateway/agents.conf` — a plain nginx include
file with one `location /agent/<name>/` block per agent. UDS file with one `location /agent/<name>/` block per agent. UDS
upstream (`http://unix:/run/hive-agent/<name>/web.sock:/`) when upstream (`http://unix:/run/hive-agent/<name>/web.sock:/`) when
`hyperhive-socket-bound` marker present; TCP loopback otherwise. `hyperhive-socket-bound` marker present; TCP loopback for agents
that haven't yet been rebuilt under the new config.
The gateway container bind-mounts `/var/lib/hyperhive/gateway/` at The gateway container bind-mounts `/var/lib/hyperhive/gateway/` at
`/run/hive-state/`; nginx includes `/run/hive-state/agents.conf`. `/run/hive-state/`; nginx includes `/run/hive-state/agents.conf`.
After each write, c0re triggers `nginx -s reload` inside the After each write, c0re triggers `nginx -s reload` inside the
gateway container from the HOST via gateway container from the HOST via
`systemd-run --machine=hive-gateway nginx -s reload`. This is `systemd-run --machine=hive-gateway --wait nginx -s reload`. This is
intentionally host-side: `IN_MOVED_TO` from an atomic rename does intentionally host-side: `IN_MOVED_TO` from an atomic rename does
not propagate across the nspawn mount-namespace boundary, so a not propagate across the nspawn mount-namespace boundary, so a
path unit inside the container would never fire (#889). path unit inside the container would never fire.
c0re regenerates `agents.conf` (and triggers a reload) on two c0re regenerates `agents.conf` (and triggers a reload) on two
triggers: every topology change (new/removed agents) and every 10s triggers: every topology change (new/removed agents) and every 10s
marker poll tick (`agent_sockets::spawn_poll`). `write()` is marker poll tick (`agent_sockets::spawn_poll`). `write()` is
idempotent — skips the rename+reload when content is unchanged. idempotent — skips the rename when content is unchanged. Failed reloads
are retried automatically on subsequent poll ticks via
Transition: agents that haven't flipped `useUnixSocket = true` get a `gateway_nginx::reload_if_pending`.
TCP loopback upstream in `agents.conf` (deterministic port from
`agent_web_port(name)`). A future cleanup will drop the TCP fallback
once every agent's flipped.
## Agent port map (`agent-ports.json`) ## Agent port map (`agent-ports.json`)

35
hive-c0re/src/agent_sockets.rs
View file

@ -30,12 +30,11 @@ pub const AGENT_SOCKET_DIR: &str = "/run/hive-agent";
pub const SOCKET_FILENAME: &str = "web.sock"; pub const SOCKET_FILENAME: &str = "web.sock";
/// Marker file the harness drops next to the socket after a /// Marker file the harness drops next to the socket after a
/// successful `bind_unix`. Presence = "this agent has opted in to /// successful `bind_unix`. Presence = "harness has bound the socket,
/// `hyperhive.web.useUnixSocket = true` and its harness has bound /// unix upstream is live"; absence = "harness hasn't started yet or
/// the socket"; absence = "the harness is still on TCP, don't /// hasn't been rebuilt under the new config — keep TCP fallback".
/// publish the unix upstream for this agent yet". Without this gate /// Without this gate the gateway would `proxy_pass` to a non-existent
/// the gateway would `proxy_pass` to a non-existent socket for every /// socket for an agent that's still starting up after a rebuild.
/// sub-agent that hasn't flipped the option yet.
/// ///
/// Renamed from `.bound` (legacy) to match the `hyperhive-` prefix /// Renamed from `.bound` (legacy) to match the `hyperhive-` prefix
/// convention for all harness-written state files. `build_map` /// convention for all harness-written state files. `build_map`
@ -70,8 +69,8 @@ pub fn socket_path_for(name: &str) -> PathBuf {
/// Includes manager and sub-agents. Filters by `READY_MARKER` /// Includes manager and sub-agents. Filters by `READY_MARKER`
/// presence: only agents whose harness has actually bound the unix /// presence: only agents whose harness has actually bound the unix
/// socket appear in the map. Without this, the gateway would /// socket appear in the map. Without this, the gateway would
/// `proxy_pass` to a non-existent socket for every sub-agent that /// `proxy_pass` to a non-existent socket for agents that haven't
/// hasn't yet flipped `hyperhive.web.useUnixSocket = true`. /// been rebuilt yet or are mid-restart.
/// ///
/// Accepts either the new `hyperhive-socket-bound` marker or the legacy /// Accepts either the new `hyperhive-socket-bound` marker or the legacy
/// `.bound` marker so existing containers keep their gateway routing /// `.bound` marker so existing containers keep their gateway routing
@ -166,12 +165,17 @@ pub fn write(names: &[String]) -> Result<()> {
} }
/// Spawn the marker poll task. Periodically re-runs `write` so the /// Spawn the marker poll task. Periodically re-runs `write` so the
/// JSON map picks up newly-bound sockets (an agent flipping /// JSON map picks up newly-bound sockets after a rebuild (harness
/// `hyperhive.web.useUnixSocket = true`, rebuilding, then having its /// drops a fresh `.bound` marker on start) without needing an explicit
/// harness drop a fresh `.bound` marker) without needing an explicit /// hook on container
/// hook on container start. `write` is idempotent (skips the rename /// start. `write` is idempotent (skips the rename when content
/// when content unchanged) so the steady-state cost is one directory /// unchanged) so the steady-state cost is one directory stat per
/// stat per agent per poll interval. /// agent per poll interval.
///
/// Also calls `gateway_nginx::reload_if_pending` on every tick to
/// retry a gateway nginx reload that may have failed on the previous
/// tick (e.g. gateway container temporarily down). This recovers
/// gateway routing without needing a manual gateway restart.
/// ///
/// Mirrors the spawn-loop shape used by `crash_watch`, /// Mirrors the spawn-loop shape used by `crash_watch`,
/// `reminder_scheduler`, etc. — the existing background-task /// `reminder_scheduler`, etc. — the existing background-task
@ -198,6 +202,9 @@ pub fn spawn_poll() {
if let Err(e) = crate::gateway_nginx::write(&names) { if let Err(e) = crate::gateway_nginx::write(&names) {
tracing::debug!(error = ?e, "gateway_nginx poll write failed"); tracing::debug!(error = ?e, "gateway_nginx poll write failed");
} }
// Retry a pending nginx reload that failed on a
// previous tick (no-op if no reload is pending).
crate::gateway_nginx::reload_if_pending();
} }
Err(e) => { Err(e) => {
tracing::debug!(error = ?e, "agent_sockets poll: failed to list agents"); tracing::debug!(error = ?e, "agent_sockets poll: failed to list agents");

51
hive-c0re/src/gateway_nginx.rs
View file

@ -8,10 +8,18 @@
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use std::fmt::Write as _; use std::fmt::Write as _;
use std::path::PathBuf; use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, Ordering};
use crate::agent_sockets; use crate::agent_sockets;
use crate::lifecycle; use crate::lifecycle;
/// Set when `write` publishes a new agents.conf; cleared when
/// `reload_gateway_nginx` submits the reload command successfully.
/// Lets `spawn_poll` retry the reload on subsequent ticks when the
/// previous attempt failed (e.g. gateway container temporarily down,
/// systemd-run not found) without re-writing the already-correct file.
static RELOAD_PENDING: AtomicBool = AtomicBool::new(false);
const HOST_CONF_PATH: &str = "/var/lib/hyperhive/gateway/agents.conf"; const HOST_CONF_PATH: &str = "/var/lib/hyperhive/gateway/agents.conf";
/// Host-side path where c0re writes the generated nginx include file. /// Host-side path where c0re writes the generated nginx include file.
@ -64,7 +72,7 @@ fn render(names: &[String], frontend_dir: Option<&str>) -> String {
let mut out = String::from( let mut out = String::from(
"# Generated by hive-c0re \u{2014} do not edit.\ "# Generated by hive-c0re \u{2014} do not edit.\
\n# Refreshed on every topology change + when agents bind/drop their unix sockets.\ \n# Refreshed on every topology change + when agents bind/drop their unix sockets.\
\n# Gateway reloads nginx automatically on each update (systemd path unit).\n", \n# Reload triggered by hive-c0re via systemd-run --machine=hive-gateway.\n",
); );
for name in names { for name in names {
let port = lifecycle::agent_web_port(name); let port = lifecycle::agent_web_port(name);
@ -184,24 +192,40 @@ pub fn write(names: &[String]) -> Result<()> {
path.display() path.display()
) )
})?; })?;
// Trigger nginx reload from the host. Ignore errors — a failed // Mark reload pending before attempting so a failed attempt is
// reload is recoverable (nginx keeps serving the previous config). // retried by the next spawn_poll tick (see `reload_if_pending`).
RELOAD_PENDING.store(true, Ordering::Relaxed);
reload_gateway_nginx(); reload_gateway_nginx();
Ok(()) Ok(())
} }
/// Retry a pending nginx reload if a previous attempt failed.
/// Called by `spawn_poll` on each tick so a transient failure
/// (gateway container temporarily down, systemd-run error) is
/// recovered automatically without requiring a new file write.
pub fn reload_if_pending() {
if RELOAD_PENDING.load(Ordering::Relaxed) {
reload_gateway_nginx();
}
}
/// Send `nginx -s reload` to the gateway container via systemd-run. /// Send `nginx -s reload` to the gateway container via systemd-run.
/// Runs non-interactively in a transient scope so it doesn't block /// Uses `--wait` so the exit code reflects whether nginx received the
/// c0re's polling loop. Best-effort: errors are logged, not bubbled. /// signal; clears `RELOAD_PENDING` on success so `reload_if_pending`
/// stops retrying. Best-effort: errors are logged, not bubbled.
fn reload_gateway_nginx() { fn reload_gateway_nginx() {
// `--machine=hive-gateway` targets the container by its nspawn // `--machine=hive-gateway` targets the container by its nspawn
// machine name (same as the nixos-container name). `--quiet` // machine name (same as the nixos-container name). `--quiet`
// suppresses the transient unit name echo. `--` separates // suppresses the transient unit name echo. `--wait` blocks until
// systemd-run args from the command. // the transient job exits so the exit code tells us whether
// `nginx -s reload` ran at all (RELOAD_PENDING is only cleared on
// success — a failed attempt is retried next tick). `--`
// separates systemd-run args from the command.
let status = std::process::Command::new("systemd-run") let status = std::process::Command::new("systemd-run")
.args([ .args([
"--machine=hive-gateway", "--machine=hive-gateway",
"--quiet", "--quiet",
"--wait",
"--", "--",
"nginx", "nginx",
"-s", "-s",
@ -210,21 +234,22 @@ fn reload_gateway_nginx() {
.status(); .status();
match status { match status {
Ok(s) if s.success() => { Ok(s) if s.success() => {
// systemd-run accepted the request; nginx reload runs // nginx -s reload ran successfully (SIGHUP sent to master).
// asynchronously inside the container and may still fail // The actual worker replacement is async but the signal was
// silently, but that's acceptable given the best-effort contract. // delivered; clear the pending flag.
tracing::debug!("systemd-run accepted gateway nginx reload request"); RELOAD_PENDING.store(false, Ordering::Relaxed);
tracing::debug!("gateway nginx reload signal sent");
} }
Ok(s) => { Ok(s) => {
tracing::warn!( tracing::warn!(
exit_code = ?s.code(), exit_code = ?s.code(),
"gateway nginx reload exited non-zero; will pick up on next restart" "gateway nginx reload exited non-zero — will retry next poll tick"
); );
} }
Err(e) => { Err(e) => {
tracing::warn!( tracing::warn!(
error = %e, error = %e,
"failed to invoke systemd-run for gateway nginx reload" "failed to invoke systemd-run for gateway nginx reload — will retry next poll tick"
); );
} }
} }

12
hive-c0re/src/main.rs
View file

@ -261,12 +261,12 @@ async fn cmd_serve(
// operator-initiated transient state. // operator-initiated transient state.
crash_watch::spawn(coord.clone()); crash_watch::spawn(coord.clone());
// Agent-sockets marker poll: re-fires `agent_sockets::write` // Agent-sockets marker poll: re-fires `agent_sockets::write`
// every 10s so the JSON picks up newly-bound `.bound` markers // and `gateway_nginx::write` every 10s so the JSON and nginx
// (a sub-agent flipping `hyperhive.web.useUnixSocket = true`, // config pick up newly-bound `.bound` markers after a rebuild.
// rebuilding, then having its harness bind the socket) without // Also retries any pending gateway nginx reload that failed on
// needing an explicit hook on each container start. write() is // the previous tick. write() is idempotent so steady-state cost
// idempotent so steady-state cost is one stat per agent per // is one stat per agent per tick.
// tick. See `docs/gateway.md::Per-agent unix-socket upstream`. // See `docs/gateway.md::Per-agent unix-socket upstream`.
agent_sockets::spawn_poll(); agent_sockets::spawn_poll();
// Reminder scheduler: drains due reminders + handles // Reminder scheduler: drains due reminders + handles
// file_path payload persistence. See reminder_scheduler.rs. // file_path payload persistence. See reminder_scheduler.rs.

37
nix/templates/harness-base.nix
View file

@ -80,24 +80,9 @@ in
default = false; default = false;
example = true; example = true;
description = '' description = ''
When `true`, set `HIVE_WEB_SOCKET=/run/hive-agent/${userName}/web.sock` Deprecated. Unix socket mode is now always enabled for all agents.
on the harness service env, which makes `web_ui::serve` bind a Setting this option to `true` has no effect and the option will be
`UnixListener` at that path instead of the legacy TCP listener removed in a future version. Safe to drop from agent configs.
on `HIVE_PORT`.
Default `false` so an agent's web UI keeps binding TCP until
the per-agent flip is explicit. Rollout shape:
1. flip one canary agent to `true` via its `agent.nix`;
2. validate the gateway's `proxy_pass http://unix:.../web.sock`
end-to-end against that canary;
3. flip remaining agents per-agent as the gateway side soaks;
4. eventually drop this option once every agent is on unix and
the TCP fallback is removed from the harness.
Sub-agents only: the manager always has `HIVE_WEB_SOCKET` set
unconditionally in the `isManager` env block, so this toggle
has no effect when `hyperhive.role = "manager"`.
''; '';
}; };
@ -1304,23 +1289,15 @@ in
HIVE_STATIC_DIR = "${config.hyperhive.frontend.mergedDist}"; HIVE_STATIC_DIR = "${config.hyperhive.frontend.mergedDist}";
HIVE_ASSETS_DIR = "${pkgs.hyperhive-assets}/share/hyperhive"; HIVE_ASSETS_DIR = "${pkgs.hyperhive-assets}/share/hyperhive";
HIVE_ROLE = config.hyperhive.role; HIVE_ROLE = config.hyperhive.role;
} # Unix-socket path for the harness web UI. All agents (sub-agents
// lib.optionalAttrs config.hyperhive.web.useUnixSocket { # and manager) always bind here; TCP fallback is removed. Path
# Per-agent unix-socket path for the web UI. When set, # matches `hive_c0re::agent_sockets::socket_path_for(name)` so
# the harness's `web_ui::serve` binds a `UnixListener` # lifecycle bind-mounts and gateway upstream config stay in sync.
# at this path instead of TCP. Path matches
# `hive_c0re::agent_sockets::socket_path_for(name)` so
# the lifecycle bind-mount and the gateway's upstream
# config all derive from the same canonical
# `/run/hive-agent/<name>/web.sock` shape.
HIVE_WEB_SOCKET = "/run/hive-agent/${userName}/web.sock"; HIVE_WEB_SOCKET = "/run/hive-agent/${userName}/web.sock";
} }
// lib.optionalAttrs isManager { // lib.optionalAttrs isManager {
# Standalone-eval fallback; meta.rs overrides at deploy time. # Standalone-eval fallback; meta.rs overrides at deploy time.
HIVE_LABEL = "root"; HIVE_LABEL = "root";
# Manager always uses a unix socket so the gateway can route
# /agent/<name>/ to it the same way it routes sub-agents.
HIVE_WEB_SOCKET = "/run/hive-agent/${userName}/web.sock";
}; };
serviceConfig = { serviceConfig = {
ExecStart = "${pkgs.hyperhive}/bin/${binary} serve"; ExecStart = "${pkgs.hyperhive}/bin/${binary} serve";