gateway: hot-reload agents.conf at runtime (#869)
Replace eval-time per-agent nginx location baking with a runtime
include file. c0re writes /var/lib/hyperhive/agents.conf (nginx
location blocks, UDS or TCP per agent) on every topology change and
on the 10s marker poll. The gateway container bind-mounts
/var/lib/hyperhive/ at /run/hive-state/ and nginx includes
/run/hive-state/agents.conf. A systemd path unit inside the container
watches the file for changes and fires `nginx -s reload` on each
atomic rename from c0re — no nixos-rebuild switch needed when agents
start, stop, or flip useUnixSocket.
- new hive-c0re/src/gateway_nginx.rs: write() + render()
- lib.rs + meta.rs + agent_sockets::spawn_poll: hook in write()
- hive-gateway.nix: drop agentPortsTable/agentSocketsTable/
agentUpstreamFor/lib.mapAttrs', add /run/hive-state bind-mount,
include directive, systemd path unit + reload service, tmpfiles
for /var/lib/hyperhive + agents.conf seed
- docs/gateway.md: update vhost table + Per-agent UDS section
This commit is contained in:
parent
f1d2063a84
commit
07434e8f50
7 changed files with 337 additions and 173 deletions
|
|
@ -7,7 +7,7 @@ Single nginx in front of every hyperhive web surface. Container `hive-gateway`,
|
||||||
| URL | vhost | upstream | source |
|
| URL | vhost | upstream | source |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| `<hive>/` | `_` (catch-all) | hive-c0re dashboard (`7000`) | always |
|
| `<hive>/` | `_` (catch-all) | hive-c0re dashboard (`7000`) | always |
|
||||||
| `<hive>/agent/<name>/` | `_` | per-agent harness on `agent_web_port(name)` | `agentPortsFile` JSON |
|
| `<hive>/agent/<name>/` | `_` | per-agent harness (UDS or TCP) | `agents.conf` (runtime-generated) |
|
||||||
| `<hive>/.well-known/matrix/{client,server}` | `_` | inline JSON (no upstream) | `matrix.enable && domain != null` |
|
| `<hive>/.well-known/matrix/{client,server}` | `_` | inline JSON (no upstream) | `matrix.enable && domain != null` |
|
||||||
| `<hive>/matrix/` (deprecated) | `_` | 301 → `matrix.<hive>/` | `matrix.gui.enable` |
|
| `<hive>/matrix/` (deprecated) | `_` | 301 → `matrix.<hive>/` | `matrix.gui.enable` |
|
||||||
| `forge.<hive>/` | `forge.<hive>` | forgejo (`3000`) | `forge.behindGateway` |
|
| `forge.<hive>/` | `forge.<hive>` | forgejo (`3000`) | `forge.behindGateway` |
|
||||||
|
|
@ -89,21 +89,27 @@ unix-domain socket as each agent opts in. The mechanism:
|
||||||
harness has actually bound the socket appear there. Without this
|
harness has actually bound the socket appear there. Without this
|
||||||
filter, the gateway would `proxy_pass` to a non-existent socket
|
filter, the gateway would `proxy_pass` to a non-existent socket
|
||||||
for every sub-agent that hasn't opted in yet.
|
for every sub-agent that hasn't opted in yet.
|
||||||
4. **Gateway side**. Reads `agent-sockets.json` at request-handling
|
4. **Gateway side**. `gateway_nginx::write` generates
|
||||||
time and routes `/agent/<name>/` to
|
`/var/lib/hyperhive/agents.conf` — a plain nginx include file with
|
||||||
`http://unix:/run/hive-agent/<name>/web.sock:/`. Whole
|
one `location /agent/<name>/` block per agent. UDS upstream
|
||||||
`/run/hive-agent/` is bind-mounted read-only into the gateway
|
(`http://unix:/run/hive-agent/<name>/web.sock:/`) when `.bound`
|
||||||
container so it can reach every published socket.
|
marker present; TCP loopback otherwise. The gateway container
|
||||||
|
bind-mounts `/var/lib/hyperhive/` at `/run/hive-state/`; nginx
|
||||||
|
includes `/run/hive-state/agents.conf`. A systemd path unit
|
||||||
|
(`hive-gateway-agents-conf.path`) inside the container watches the
|
||||||
|
file and fires `nginx -s reload` on every atomic rename from c0re
|
||||||
|
— no `nixos-rebuild` needed (#869).
|
||||||
|
|
||||||
c0re re-fires `agent_sockets::write` every 10s so newly-bound
|
c0re regenerates `agents.conf` (and fires the path unit → reload) on
|
||||||
markers get picked up without needing a container-start hook in
|
two triggers: every topology change (new/removed agents) and every
|
||||||
every lifecycle path. `write()` is idempotent: steady-state cost is
|
10s marker poll tick (`agent_sockets::spawn_poll`). `write()` is
|
||||||
one stat per agent per tick.
|
idempotent — skips the rename when content is unchanged so the path
|
||||||
|
unit doesn't fire spuriously.
|
||||||
|
|
||||||
Transition: agents that haven't flipped `useUnixSocket = true` still
|
Transition: agents that haven't flipped `useUnixSocket = true` get a
|
||||||
appear in `agent-ports.json` (the legacy TCP map) and the gateway
|
TCP loopback upstream in `agents.conf` (deterministic port from
|
||||||
falls back to TCP for them. A future cleanup will drop the TCP map +
|
`agent_web_port(name)`). A future cleanup will drop the TCP fallback
|
||||||
the harness's TCP bind once every agent's flipped.
|
once every agent's flipped.
|
||||||
|
|
||||||
## Dashboard link shape (gateway vs direct)
|
## Dashboard link shape (gateway vs direct)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,11 @@
|
||||||
//! `/var/lib/hyperhive/agent-ports.json` writer. Legacy TCP map for
|
//! `/var/lib/hyperhive/agent-ports.json` writer. Port map for
|
||||||
//! per-agent `/agent/<name>/` routing — the unix-socket replacement
|
//! per-agent `/agent/<name>/` TCP routing. Written alongside
|
||||||
//! lives in `agent_sockets.rs`. The gateway reads this JSON at
|
//! `agents.conf` (see `gateway_nginx.rs`) on every topology change;
|
||||||
//! request-handling time rather than at gateway build time, so a
|
//! `gateway_nginx::render` reads it indirectly via
|
||||||
//! `nixos-container update` of the gateway isn't needed every time
|
//! `lifecycle::agent_web_port` to populate TCP upstreams for agents
|
||||||
//! an agent spawns / moves / destroys.
|
//! that haven't opted in to unix-socket mode yet. Also kept as a
|
||||||
|
//! human-readable audit file — `cat agent-ports.json` shows every
|
||||||
|
//! registered sub-agent and its deterministic port assignment.
|
||||||
//!
|
//!
|
||||||
//! Shape (flat object keyed by logical agent name → web port):
|
//! Shape (flat object keyed by logical agent name → web port):
|
||||||
//!
|
//!
|
||||||
|
|
|
||||||
|
|
@ -181,6 +181,14 @@ pub fn spawn_poll() {
|
||||||
if let Err(e) = write(&names) {
|
if let Err(e) = write(&names) {
|
||||||
tracing::debug!(error = ?e, "agent_sockets poll write failed");
|
tracing::debug!(error = ?e, "agent_sockets poll write failed");
|
||||||
}
|
}
|
||||||
|
// Regenerate the gateway nginx include whenever
|
||||||
|
// socket readiness changes — the upstream
|
||||||
|
// selection (UDS vs TCP) depends on .bound markers
|
||||||
|
// which change independently of topology. Write is
|
||||||
|
// idempotent; skips rename when nothing changed.
|
||||||
|
if let Err(e) = crate::gateway_nginx::write(&names) {
|
||||||
|
tracing::debug!(error = ?e, "gateway_nginx poll write failed");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
tracing::debug!(error = ?e, "agent_sockets poll: failed to list agents");
|
tracing::debug!(error = ?e, "agent_sockets poll: failed to list agents");
|
||||||
|
|
|
||||||
205
hive-c0re/src/gateway_nginx.rs
Normal file
205
hive-c0re/src/gateway_nginx.rs
Normal file
|
|
@ -0,0 +1,205 @@
|
||||||
|
//! Runtime nginx include-file generator for the gateway's per-agent
|
||||||
|
//! `/agent/<name>/` location blocks (#869).
|
||||||
|
//!
|
||||||
|
//! Writes `/var/lib/hyperhive/agents.conf` on every topology change.
|
||||||
|
//! The gateway container bind-mounts the whole `/var/lib/hyperhive/`
|
||||||
|
//! directory at `/run/hive-state/` and nginx includes
|
||||||
|
//! `/run/hive-state/agents.conf`. A systemd path unit inside the
|
||||||
|
//! gateway container watches the file and triggers `nginx -s reload`
|
||||||
|
//! on every atomic rename — no `nixos-rebuild switch` needed when
|
||||||
|
//! agents start, stop, or flip `useUnixSocket`.
|
||||||
|
//!
|
||||||
|
//! Upstream selection mirrors `agent_sockets::build_map`: an agent
|
||||||
|
//! gets a UDS upstream when its `.bound` marker exists (harness has
|
||||||
|
//! bound the unix socket); otherwise falls back to the deterministic
|
||||||
|
//! TCP port from `lifecycle::agent_web_port`. Proxy headers are
|
||||||
|
//! emitted in full so the generated file is self-contained nginx
|
||||||
|
//! config — no dependency on which `recommendedProxySettings` knobs
|
||||||
|
//! the host config has on.
|
||||||
|
//!
|
||||||
|
//! `write()` is idempotent: if the rendered body equals what's already
|
||||||
|
//! on disk, the rename is skipped and the path unit doesn't fire.
|
||||||
|
//! Same atomic `<path>.tmp` + `rename()` shape as `agent_ports` /
|
||||||
|
//! `agent_sockets` — a crashing c0re process never leaves a partial
|
||||||
|
//! file the gateway's nginx would fail to parse.
|
||||||
|
|
||||||
|
use anyhow::{Context, Result};
|
||||||
|
use std::fmt::Write as _;
|
||||||
|
use std::path::PathBuf;
|
||||||
|
|
||||||
|
use crate::agent_sockets;
|
||||||
|
use crate::lifecycle::{self, MANAGER_NAME};
|
||||||
|
|
||||||
|
const HOST_CONF_PATH: &str = "/var/lib/hyperhive/agents.conf";
|
||||||
|
|
||||||
|
/// Host-side path where c0re writes the generated nginx include file.
|
||||||
|
/// The gateway container bind-mounts `/var/lib/hyperhive/` at
|
||||||
|
/// `/run/hive-state/` so nginx inside can read it at
|
||||||
|
/// `/run/hive-state/agents.conf`.
|
||||||
|
#[must_use]
|
||||||
|
pub fn host_conf_path() -> PathBuf {
|
||||||
|
PathBuf::from(HOST_CONF_PATH)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Nginx proxy headers present in every per-agent location block.
|
||||||
|
/// `$connection_upgrade` is defined in the http context by the NixOS
|
||||||
|
/// nginx module when `recommendedProxySettings = true` (which the
|
||||||
|
/// gateway always sets). `X-Forwarded-Prefix` is per-location (set
|
||||||
|
/// below in the render loop). Kept as a standalone const so it's
|
||||||
|
/// visible to tests without repeating the text.
|
||||||
|
const PROXY_HEADER_BLOCK: &str = " proxy_http_version 1.1;
|
||||||
|
proxy_set_header Upgrade $http_upgrade;
|
||||||
|
proxy_set_header Connection $connection_upgrade;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;";
|
||||||
|
|
||||||
|
/// Render the nginx include body for `names`. Manager is filtered out.
|
||||||
|
/// Each remaining agent gets a `location /agent/<name>/` block whose
|
||||||
|
/// `proxy_pass` targets the unix socket when the `.bound` marker
|
||||||
|
/// exists, or TCP loopback otherwise.
|
||||||
|
///
|
||||||
|
/// Output is deterministic for the same (sorted) set of names +
|
||||||
|
/// `.bound` state: no timestamps, no UUIDs. `BTreeMap` would give
|
||||||
|
/// alphabetical order; we rely on the caller to pass sorted names if
|
||||||
|
/// they care about diff stability, but the gateway treats the blocks
|
||||||
|
/// as unordered by nginx's longest-prefix-match rules so ordering only
|
||||||
|
/// affects human readability.
|
||||||
|
fn render(names: &[String]) -> String {
|
||||||
|
let mut out = String::from(
|
||||||
|
"# Generated by hive-c0re \u{2014} do not edit.\
|
||||||
|
\n# Refreshed on every topology change + when agents bind/drop their unix sockets.\
|
||||||
|
\n# Gateway reloads nginx automatically on each update (systemd path unit).\n",
|
||||||
|
);
|
||||||
|
for name in names {
|
||||||
|
if name == MANAGER_NAME {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let port = lifecycle::agent_web_port(name);
|
||||||
|
let upstream = if agent_sockets::ready_marker_for(name).exists() {
|
||||||
|
format!(
|
||||||
|
"http://unix:{}:/",
|
||||||
|
agent_sockets::socket_path_for(name).display()
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
format!("http://127.0.0.1:{port}/")
|
||||||
|
};
|
||||||
|
let _ = write!(
|
||||||
|
out,
|
||||||
|
"\nlocation /agent/{name}/ {{\n\
|
||||||
|
{PROXY_HEADER_BLOCK}\n\
|
||||||
|
\n proxy_pass {upstream};\n\
|
||||||
|
\n proxy_set_header X-Forwarded-Prefix /agent/{name};\n\
|
||||||
|
\n proxy_buffering off;\n\
|
||||||
|
\n proxy_read_timeout 1d;\n\
|
||||||
|
\n proxy_intercept_errors on;\n\
|
||||||
|
\n error_page 502 503 504 = /__hive_agent_unreachable;\n\
|
||||||
|
}}\n",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Atomically write the nginx include file for `names` to
|
||||||
|
/// [`host_conf_path()`]. Skips the rename when the rendered body
|
||||||
|
/// matches what's already on disk (idempotent; avoids spurious
|
||||||
|
/// gateway reloads on a quiet tick). On a fresh install where the
|
||||||
|
/// file doesn't exist yet, writes an empty-but-valid config so nginx
|
||||||
|
/// can start before any agents have registered.
|
||||||
|
pub fn write(names: &[String]) -> Result<()> {
|
||||||
|
let body = render(names);
|
||||||
|
let path = host_conf_path();
|
||||||
|
if std::fs::read_to_string(&path).ok().as_deref() == Some(&body) {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
if let Some(parent) = path.parent() {
|
||||||
|
std::fs::create_dir_all(parent)
|
||||||
|
.with_context(|| format!("create dir {}", parent.display()))?;
|
||||||
|
}
|
||||||
|
let tmp = path.with_extension("conf.tmp");
|
||||||
|
std::fs::write(&tmp, &body)
|
||||||
|
.with_context(|| format!("write tmp {}", tmp.display()))?;
|
||||||
|
std::fs::rename(&tmp, &path).with_context(|| {
|
||||||
|
format!(
|
||||||
|
"rename {} -> {} (atomic publish)",
|
||||||
|
tmp.display(),
|
||||||
|
path.display()
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn render_empty_is_header_only() {
|
||||||
|
let body = render(&[]);
|
||||||
|
assert!(body.starts_with("# Generated by hive-c0re"));
|
||||||
|
// No location blocks when no agents.
|
||||||
|
assert!(!body.contains("location"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn render_filters_manager() {
|
||||||
|
let names: Vec<String> = [MANAGER_NAME, "iris"]
|
||||||
|
.iter()
|
||||||
|
.map(|s| (*s).to_owned())
|
||||||
|
.collect();
|
||||||
|
// Use build_map_with so we don't need real .bound files.
|
||||||
|
// For render, we call it directly with names.
|
||||||
|
let body = render(&names);
|
||||||
|
assert!(!body.contains(&format!("/agent/{MANAGER_NAME}/")));
|
||||||
|
assert!(body.contains("/agent/iris/"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn render_tcp_upstream_when_no_bound_marker() {
|
||||||
|
// No .bound file on disk → falls back to TCP loopback.
|
||||||
|
let names = vec!["iris".to_owned()];
|
||||||
|
let body = render(&names);
|
||||||
|
let port = lifecycle::agent_web_port("iris");
|
||||||
|
assert!(
|
||||||
|
body.contains(&format!("proxy_pass http://127.0.0.1:{port}/")),
|
||||||
|
"expected TCP upstream for iris, got:\n{body}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn render_includes_proxy_headers() {
|
||||||
|
let names = vec!["iris".to_owned()];
|
||||||
|
let body = render(&names);
|
||||||
|
// All required proxy headers present (WebSocket upgrade + forwarded-for).
|
||||||
|
assert!(body.contains("proxy_http_version 1.1"));
|
||||||
|
assert!(body.contains("proxy_set_header Upgrade $http_upgrade"));
|
||||||
|
assert!(body.contains("proxy_set_header X-Forwarded-For"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn render_includes_forwarded_prefix() {
|
||||||
|
let names = vec!["atlas".to_owned()];
|
||||||
|
let body = render(&names);
|
||||||
|
assert!(
|
||||||
|
body.contains("X-Forwarded-Prefix /agent/atlas"),
|
||||||
|
"expected X-Forwarded-Prefix for atlas, got:\n{body}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn render_includes_error_pages() {
|
||||||
|
let names = vec!["iris".to_owned()];
|
||||||
|
let body = render(&names);
|
||||||
|
assert!(body.contains("proxy_intercept_errors on"));
|
||||||
|
assert!(body.contains("__hive_agent_unreachable"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn render_includes_buffering_and_timeout() {
|
||||||
|
let names = vec!["iris".to_owned()];
|
||||||
|
let body = render(&names);
|
||||||
|
assert!(body.contains("proxy_buffering off"));
|
||||||
|
assert!(body.contains("proxy_read_timeout 1d"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -16,6 +16,7 @@ pub mod actions;
|
||||||
pub mod agent_ports;
|
pub mod agent_ports;
|
||||||
pub mod agent_server;
|
pub mod agent_server;
|
||||||
pub mod agent_sockets;
|
pub mod agent_sockets;
|
||||||
|
pub mod gateway_nginx;
|
||||||
pub mod approvals;
|
pub mod approvals;
|
||||||
pub mod auto_update;
|
pub mod auto_update;
|
||||||
pub mod broker;
|
pub mod broker;
|
||||||
|
|
|
||||||
|
|
@ -126,6 +126,16 @@ pub async fn sync_agents(
|
||||||
tracing::warn!(error = ?e, "agent_sockets::write failed (non-fatal)");
|
tracing::warn!(error = ?e, "agent_sockets::write failed (non-fatal)");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Refresh /var/lib/hyperhive/agents.conf — the nginx include file
|
||||||
|
// the gateway picks up at runtime without needing a
|
||||||
|
// nixos-rebuild. The gateway container bind-mounts
|
||||||
|
// /var/lib/hyperhive/ and a systemd path unit fires
|
||||||
|
// `nginx -s reload` when this file changes (#869). Same
|
||||||
|
// best-effort + non-fatal shape.
|
||||||
|
if let Err(e) = crate::gateway_nginx::write(&agent_names) {
|
||||||
|
tracing::warn!(error = ?e, "gateway_nginx::write failed (non-fatal)");
|
||||||
|
}
|
||||||
|
|
||||||
if initial {
|
if initial {
|
||||||
git(&dir, &["init", "--initial-branch=main"]).await?;
|
git(&dir, &["init", "--initial-branch=main"]).await?;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -11,43 +11,6 @@ let
|
||||||
forgeCfg = config.services.hyperhive.forge;
|
forgeCfg = config.services.hyperhive.forge;
|
||||||
networkCfg = config.services.hyperhive.network;
|
networkCfg = config.services.hyperhive.network;
|
||||||
|
|
||||||
# Per-agent port table for `/agent/<name>/` routing. C0re writes
|
|
||||||
# this JSON on every topology change; gateway reads at deploy time.
|
|
||||||
# Missing file → empty map → no per-agent routes (graceful default).
|
|
||||||
# See `docs/gateway.md` for the discovery + rebuild flow.
|
|
||||||
agentPortsTable =
|
|
||||||
if cfg.agentPortsFile == null || !builtins.pathExists cfg.agentPortsFile then
|
|
||||||
{ }
|
|
||||||
else
|
|
||||||
builtins.fromJSON (builtins.readFile cfg.agentPortsFile);
|
|
||||||
|
|
||||||
# Per-agent unix-socket table for `/agent/<name>/` UDS upstream
|
|
||||||
# (#784 phase 2 step 3). C0re writes this JSON alongside
|
|
||||||
# agent-ports.json; gateway reads at deploy time. Per-agent the
|
|
||||||
# entry wins over the TCP port. Missing entry (or missing file)
|
|
||||||
# → fall back to the TCP port. See
|
|
||||||
# `docs/gateway.md::Per-agent UDS upstream (#784)`.
|
|
||||||
agentSocketsTable =
|
|
||||||
if cfg.agentSocketsFile == null || !builtins.pathExists cfg.agentSocketsFile then
|
|
||||||
{ }
|
|
||||||
else
|
|
||||||
builtins.fromJSON (builtins.readFile cfg.agentSocketsFile);
|
|
||||||
|
|
||||||
# Resolve a per-agent upstream URL. Socket entry wins ONLY when the
|
|
||||||
# socket file actually exists at eval time — guards against agents
|
|
||||||
# that have an `agent-sockets.json` entry from c0re's blanket emit
|
|
||||||
# but haven't actually flipped `hyperhive.web.useUnixSocket = true`
|
|
||||||
# (their harness still binds TCP only, so a UDS upstream would 502).
|
|
||||||
# Falls back to the TCP loopback otherwise. Once c0re ships the
|
|
||||||
# `.bound` marker filter (#784 step 2d follow-up), the path-exists
|
|
||||||
# check becomes redundant but harmless; step 4 drops it entirely.
|
|
||||||
agentUpstreamFor =
|
|
||||||
name: port:
|
|
||||||
if agentSocketsTable ? ${name} && builtins.pathExists agentSocketsTable.${name} then
|
|
||||||
"http://unix:${agentSocketsTable.${name}}:/"
|
|
||||||
else
|
|
||||||
"http://127.0.0.1:${toString port}/";
|
|
||||||
|
|
||||||
# Static error pages for `/agent/<name>/` mishaps (#755). Mara's
|
# Static error pages for `/agent/<name>/` mishaps (#755). Mara's
|
||||||
# call: useful pages instead of nginx's default 404/502 for routes
|
# call: useful pages instead of nginx's default 404/502 for routes
|
||||||
# we've already special-cased. See `docs/gateway.md::Per-agent
|
# we've already special-cased. See `docs/gateway.md::Per-agent
|
||||||
|
|
@ -71,7 +34,7 @@ let
|
||||||
<body>
|
<body>
|
||||||
<h1>◆ agent not found</h1>
|
<h1>◆ agent not found</h1>
|
||||||
<p>No agent matches the requested <code>/agent/<name>/</code> path on this hive.</p>
|
<p>No agent matches the requested <code>/agent/<name>/</code> path on this hive.</p>
|
||||||
<p>Operator: check the agent name in <a href="/">the dashboard</a> — the gateway picks up new agents on the next <code>nixos-rebuild switch</code>.</p>
|
<p>Operator: check the agent name in <a href="/">the dashboard</a>.</p>
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
EOF
|
EOF
|
||||||
|
|
@ -237,75 +200,6 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
agentPortsFile = lib.mkOption {
|
|
||||||
type = lib.types.nullOr lib.types.path;
|
|
||||||
default = "/var/lib/hyperhive/agent-ports.json";
|
|
||||||
example = "/var/lib/hyperhive/agent-ports.json";
|
|
||||||
description = ''
|
|
||||||
Path to a JSON file mapping sub-agent names to their web ports
|
|
||||||
for `/agent/<name>/` routing through the gateway (#15 v0).
|
|
||||||
Shape: `{ "<name>": <port>, ... }`. Written by hive-c0re on
|
|
||||||
every topology change (the rust side knows the canonical port
|
|
||||||
allocation via `lifecycle::agent_web_port`; the gateway just
|
|
||||||
reads what it's told).
|
|
||||||
|
|
||||||
For each `<name>: <port>` entry, the gateway adds a
|
|
||||||
`location /agent/<name>/` block that `proxy_pass`es to
|
|
||||||
`http://127.0.0.1:<port>/`. Empty / missing file → no
|
|
||||||
per-agent routes generated → gateway falls back to its pre-#15
|
|
||||||
shape (just `/` + matrix surfaces).
|
|
||||||
|
|
||||||
**Purely additive**: the old `http://<host>:<port>/` direct
|
|
||||||
reach keeps working in parallel; this just gives the operator
|
|
||||||
a single-origin route. Manager isn't included in the map (no
|
|
||||||
per-agent prefix needed; manager already gets the `/` route
|
|
||||||
via the c0re upstream block).
|
|
||||||
|
|
||||||
Set to `null` to disable per-agent routing entirely without
|
|
||||||
creating the file. Set to a custom path if the operator's c0re
|
|
||||||
writes the table elsewhere.
|
|
||||||
|
|
||||||
**Rebuild trigger**: the gateway container picks up new entries
|
|
||||||
on the next `nixos-rebuild switch` (or `hivectl gateway-sync`
|
|
||||||
if that helper lands). c0re writes are not auto-applied to a
|
|
||||||
running gateway — see the follow-up in #15 for runtime nginx
|
|
||||||
include + reload + eventual per-agent unix sockets.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
agentSocketsFile = lib.mkOption {
|
|
||||||
type = lib.types.nullOr lib.types.path;
|
|
||||||
default = "/var/lib/hyperhive/agent-sockets.json";
|
|
||||||
example = "/var/lib/hyperhive/agent-sockets.json";
|
|
||||||
description = ''
|
|
||||||
Path to a JSON file mapping sub-agent names to their
|
|
||||||
per-agent unix-socket paths for `/agent/<name>/` UDS upstream
|
|
||||||
routing (#784 phase 2 step 3). Shape:
|
|
||||||
`{ "<name>": "/run/hive-agent/<name>/web.sock", ... }`.
|
|
||||||
Written by hive-c0re alongside `agentPortsFile` on every
|
|
||||||
topology change (`hive_c0re::agent_sockets::write`;
|
|
||||||
path-shape derives from
|
|
||||||
`agent_sockets::socket_path_for(name)`).
|
|
||||||
|
|
||||||
Per-agent, the socket entry wins over the TCP port: when an
|
|
||||||
agent appears in this map, the gateway's `proxy_pass` for
|
|
||||||
that agent's `/agent/<name>/` location targets
|
|
||||||
`http://unix:<path>:/` instead of `http://127.0.0.1:<port>/`.
|
|
||||||
Agents that haven't opted in (no `HIVE_WEB_SOCKET` set,
|
|
||||||
no entry in the JSON, or both files unset) fall back to
|
|
||||||
TCP via `agentPortsFile`. Coexists with the TCP map during
|
|
||||||
the rollout — eventually drops `agentPortsFile` entirely
|
|
||||||
when every agent's flipped (#784 step 4).
|
|
||||||
|
|
||||||
Set to `null` to skip UDS upstreams entirely (gateway uses
|
|
||||||
TCP for every agent regardless of what hive-c0re writes).
|
|
||||||
|
|
||||||
**Bind-mount requirement**: when this is enabled the gateway
|
|
||||||
container needs `/run/hive-agent/` bind-mounted from the
|
|
||||||
host. Handled automatically by `containers.hive-gateway`
|
|
||||||
below when at least one socket entry exists.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
|
|
||||||
config = lib.mkIf cfg.enable {
|
config = lib.mkIf cfg.enable {
|
||||||
|
|
@ -320,14 +214,21 @@ in
|
||||||
}
|
}
|
||||||
];
|
];
|
||||||
|
|
||||||
# Ensure the per-agent UDS bind-mount source exists at host boot,
|
# Ensure bind-mount sources exist at host boot before the gateway
|
||||||
# before the gateway container's first start. nspawn would
|
# container's first start. nspawn would auto-create missing dirs
|
||||||
# auto-create an empty dir if missing (argus 🟡 on #829), but a
|
# (argus 🟡 on #829), but tmpfiles rules make the intent explicit
|
||||||
# tmpfiles rule makes the intent explicit and dodges the
|
# and cover the fresh-boot window before c0re has run.
|
||||||
# fresh-boot-before-any-agent-spawn window where the dir wouldn't
|
#
|
||||||
# exist yet from c0re's per-agent `set_nspawn_flags` mkdir chain.
|
# /run/hive-agent — per-agent UDS socket dir, written by c0re's
|
||||||
|
# set_nspawn_flags when agents start.
|
||||||
|
# /var/lib/hyperhive — hyperhive state dir, created by c0re on
|
||||||
|
# first run. Also pre-seed agents.conf with an empty-but-valid
|
||||||
|
# header so nginx can start + include the file before c0re writes
|
||||||
|
# its first real content (f = create-if-absent, no overwrite).
|
||||||
systemd.tmpfiles.rules = [
|
systemd.tmpfiles.rules = [
|
||||||
"d /run/hive-agent 0755 root root - -"
|
"d /run/hive-agent 0755 root root - -"
|
||||||
|
"d /var/lib/hyperhive 0755 root root - -"
|
||||||
|
"f /var/lib/hyperhive/agents.conf 0644 root root - # Generated by hive-c0re — do not edit.\n"
|
||||||
];
|
];
|
||||||
|
|
||||||
containers.hive-gateway = {
|
containers.hive-gateway = {
|
||||||
|
|
@ -339,16 +240,25 @@ in
|
||||||
# layer that matters.
|
# layer that matters.
|
||||||
privateNetwork = false;
|
privateNetwork = false;
|
||||||
# Bind-mount the per-agent socket dir so nginx inside the gateway
|
# Bind-mount the per-agent socket dir so nginx inside the gateway
|
||||||
# container can `connect(2)` to the UDS upstreams hive-c0re
|
# container can `connect(2)` to the UDS upstreams (#784 step 3).
|
||||||
# publishes in `agent-sockets.json` (#784 phase 2 step 3).
|
# Read-only (we just connect; harness writes the socket inside
|
||||||
# Read-only (we don't bind anything here; just connect). Mount
|
# the agent's own container). Host-side dir is pre-created by a
|
||||||
# is unconditional but inert when no agents have opted in:
|
# tmpfiles rule so nspawn always finds a source at boot.
|
||||||
# agent-sockets.json missing/empty → `agentSocketsTable = {}`
|
|
||||||
# → every per-agent location uses the TCP fallback.
|
|
||||||
bindMounts."/run/hive-agent" = {
|
bindMounts."/run/hive-agent" = {
|
||||||
hostPath = "/run/hive-agent";
|
hostPath = "/run/hive-agent";
|
||||||
isReadOnly = true;
|
isReadOnly = true;
|
||||||
};
|
};
|
||||||
|
# Bind-mount the hyperhive state dir so nginx can include the
|
||||||
|
# runtime-generated agents.conf. Read-only; c0re writes
|
||||||
|
# /var/lib/hyperhive/agents.conf on the host and the systemd
|
||||||
|
# path unit inside the container triggers nginx -s reload on
|
||||||
|
# each atomic rename (#869). Pre-created by a tmpfiles rule so
|
||||||
|
# nspawn always finds the source at boot (c0re also writes it
|
||||||
|
# on first startup, but the container may start before c0re).
|
||||||
|
bindMounts."/run/hive-state" = {
|
||||||
|
hostPath = "/var/lib/hyperhive";
|
||||||
|
isReadOnly = true;
|
||||||
|
};
|
||||||
config =
|
config =
|
||||||
{ pkgs, ... }:
|
{ pkgs, ... }:
|
||||||
let
|
let
|
||||||
|
|
@ -397,8 +307,7 @@ in
|
||||||
publicScheme = if cfg.selfSignedTls then "https" else "http";
|
publicScheme = if cfg.selfSignedTls then "https" else "http";
|
||||||
publicPort = if cfg.selfSignedTls then cfg.httpsPort else cfg.port;
|
publicPort = if cfg.selfSignedTls then cfg.httpsPort else cfg.port;
|
||||||
publicPortDefault = if cfg.selfSignedTls then 443 else 80;
|
publicPortDefault = if cfg.selfSignedTls then 443 else 80;
|
||||||
publicPortSuffix =
|
publicPortSuffix = if publicPort == publicPortDefault then "" else ":${toString publicPort}";
|
||||||
if publicPort == publicPortDefault then "" else ":${toString publicPort}";
|
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
system.stateVersion = "26.05";
|
system.stateVersion = "26.05";
|
||||||
|
|
@ -479,6 +388,38 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
# Watch /run/hive-state/agents.conf (bind-mounted from the
|
||||||
|
# host's /var/lib/hyperhive/agents.conf) for changes and
|
||||||
|
# trigger an nginx reload when c0re atomically renames a new
|
||||||
|
# version into place (#869). PathChanged fires on
|
||||||
|
# IN_CLOSE_WRITE + IN_MOVED_TO, so the atomic rename c0re
|
||||||
|
# uses (write .conf.tmp → rename) wakes the path unit.
|
||||||
|
# The reload is a no-op if the new config is identical —
|
||||||
|
# gateway_nginx::write skips the rename when content is
|
||||||
|
# unchanged, so the path unit doesn't fire at all on quiet
|
||||||
|
# ticks.
|
||||||
|
systemd.paths.hive-gateway-agents-conf = {
|
||||||
|
wantedBy = [ "nginx.service" ];
|
||||||
|
after = [ "nginx.service" ];
|
||||||
|
pathConfig = {
|
||||||
|
PathChanged = "/run/hive-state/agents.conf";
|
||||||
|
Unit = "hive-gateway-nginx-reload.service";
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
systemd.services.hive-gateway-nginx-reload = {
|
||||||
|
description = "Reload nginx after agents.conf change (#869)";
|
||||||
|
# Don't block any target — fires only when the path unit
|
||||||
|
# triggers it.
|
||||||
|
serviceConfig = {
|
||||||
|
Type = "oneshot";
|
||||||
|
# nginx -s reload sends SIGHUP to the master process via
|
||||||
|
# the pid file. Runs as root inside the container (pid 1
|
||||||
|
# is the nspawn init; nginx master starts as root).
|
||||||
|
ExecStart = "/run/current-system/sw/bin/nginx -s reload";
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
services.nginx = {
|
services.nginx = {
|
||||||
enable = true;
|
enable = true;
|
||||||
recommendedProxySettings = true;
|
recommendedProxySettings = true;
|
||||||
|
|
@ -551,38 +492,17 @@ in
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
//
|
|
||||||
# Per-agent UIs (#15 v0; UDS upstream #784 step 3).
|
|
||||||
# One `/agent/<name>/` block per entry in
|
|
||||||
# `agentPortsTable`. `agentUpstreamFor` resolves
|
|
||||||
# to `http://unix:<path>:/` when the agent has
|
|
||||||
# opted in via `hyperhive.web.useUnixSocket` (and
|
|
||||||
# appears in `agentSocketsTable`); otherwise
|
|
||||||
# `http://127.0.0.1:<port>/`. Trailing-slash pair
|
|
||||||
# strips the prefix; `X-Forwarded-Prefix` lets the
|
|
||||||
# harness build absolute URLs when relative isn't
|
|
||||||
# enough. `proxy_intercept_errors` + `error_page` rewrite
|
|
||||||
# upstream 502/503/504 to `unreachable.html` (#755).
|
|
||||||
lib.mapAttrs' (name: port: {
|
|
||||||
name = "/agent/${name}/";
|
|
||||||
value = {
|
|
||||||
proxyPass = agentUpstreamFor name port;
|
|
||||||
proxyWebsockets = true;
|
|
||||||
extraConfig = ''
|
|
||||||
proxy_set_header X-Forwarded-Prefix /agent/${name};
|
|
||||||
proxy_buffering off;
|
|
||||||
proxy_read_timeout 1d;
|
|
||||||
proxy_intercept_errors on;
|
|
||||||
error_page 502 503 504 = /__hive_agent_unreachable;
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
}) agentPortsTable
|
|
||||||
//
|
//
|
||||||
# `/agent/` catch-all (#755): hits when an operator
|
# `/agent/` catch-all (#755): hits when an operator
|
||||||
# requests `/agent/<unknown>/...` — a name not in
|
# requests `/agent/<unknown>/...`. Without this the
|
||||||
# `agentPortsTable`. Without this it falls through to
|
# request falls through to `/` (c0re dashboard) and
|
||||||
# `/` (c0re dashboard upstream) which returns 404
|
# returns 404 with no useful context. Custom 404
|
||||||
# with no useful context. Custom 404 page instead.
|
# page instead. Per-agent `location /agent/<name>/`
|
||||||
|
# blocks live in `/run/hive-state/agents.conf` —
|
||||||
|
# nginx picks them up via the `include` in
|
||||||
|
# `extraConfig` below; the catch-all only matches
|
||||||
|
# names that aren't in that file (nginx longest-
|
||||||
|
# prefix-match: `/agent/atlas/` beats `/agent/`).
|
||||||
{
|
{
|
||||||
"/agent/" = {
|
"/agent/" = {
|
||||||
extraConfig = ''
|
extraConfig = ''
|
||||||
|
|
@ -624,6 +544,18 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
# Per-agent location blocks, generated at runtime by
|
||||||
|
# hive-c0re and written to /var/lib/hyperhive/agents.conf
|
||||||
|
# on the host. The bind-mount at /run/hive-state/ exposes
|
||||||
|
# that file here. nginx parses `include` at config-load
|
||||||
|
# time so a reload (triggered by the hive-gateway-nginx-
|
||||||
|
# reload path unit when agents.conf changes) picks up new
|
||||||
|
# or removed agents without a nixos-rebuild. nginx's
|
||||||
|
# longest-prefix-match rule ensures `/agent/<name>/` from
|
||||||
|
# this file beats the `/agent/` catch-all above (#869).
|
||||||
|
extraConfig = ''
|
||||||
|
include /run/hive-state/agents.conf;
|
||||||
|
'';
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
//
|
//
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue