gateway: hot-reload agents.conf at runtime (#869)
Replace eval-time per-agent nginx location baking with a runtime
include file. c0re writes /var/lib/hyperhive/agents.conf (nginx
location blocks, UDS or TCP per agent) on every topology change and
on the 10s marker poll. The gateway container bind-mounts
/var/lib/hyperhive/ at /run/hive-state/ and nginx includes
/run/hive-state/agents.conf. A systemd path unit inside the container
watches the file for changes and fires `nginx -s reload` on each
atomic rename from c0re — no nixos-rebuild switch needed when agents
start, stop, or flip useUnixSocket.
- new hive-c0re/src/gateway_nginx.rs: write() + render()
- lib.rs + meta.rs + agent_sockets::spawn_poll: hook in write()
- hive-gateway.nix: drop agentPortsTable/agentSocketsTable/
agentUpstreamFor/lib.mapAttrs', add /run/hive-state bind-mount,
include directive, systemd path unit + reload service, tmpfiles
for /var/lib/hyperhive + agents.conf seed
- docs/gateway.md: update vhost table + Per-agent UDS section
This commit is contained in:
atlas
mara
parent
f1d2063a84
commit
07434e8f50
7 changed files with 337 additions and 173 deletions
|
|
@ -7,7 +7,7 @@ Single nginx in front of every hyperhive web surface. Container `hive-gateway`,
|
|||
| URL | vhost | upstream | source |
|
||||
| --- | --- | --- | --- |
|
||||
| `<hive>/` | `_` (catch-all) | hive-c0re dashboard (`7000`) | always |
|
||||
| `<hive>/agent/<name>/` | `_` | per-agent harness on `agent_web_port(name)` | `agentPortsFile` JSON |
|
||||
| `<hive>/agent/<name>/` | `_` | per-agent harness (UDS or TCP) | `agents.conf` (runtime-generated) |
|
||||
| `<hive>/.well-known/matrix/{client,server}` | `_` | inline JSON (no upstream) | `matrix.enable && domain != null` |
|
||||
| `<hive>/matrix/` (deprecated) | `_` | 301 → `matrix.<hive>/` | `matrix.gui.enable` |
|
||||
| `forge.<hive>/` | `forge.<hive>` | forgejo (`3000`) | `forge.behindGateway` |
|
||||
|
|
@ -89,21 +89,27 @@ unix-domain socket as each agent opts in. The mechanism:
|
|||
harness has actually bound the socket appear there. Without this
|
||||
filter, the gateway would `proxy_pass` to a non-existent socket
|
||||
for every sub-agent that hasn't opted in yet.
|
||||
4. **Gateway side**. Reads `agent-sockets.json` at request-handling
|
||||
time and routes `/agent/<name>/` to
|
||||
`http://unix:/run/hive-agent/<name>/web.sock:/`. Whole
|
||||
`/run/hive-agent/` is bind-mounted read-only into the gateway
|
||||
container so it can reach every published socket.
|
||||
4. **Gateway side**. `gateway_nginx::write` generates
|
||||
`/var/lib/hyperhive/agents.conf` — a plain nginx include file with
|
||||
one `location /agent/<name>/` block per agent. UDS upstream
|
||||
(`http://unix:/run/hive-agent/<name>/web.sock:/`) when `.bound`
|
||||
marker present; TCP loopback otherwise. The gateway container
|
||||
bind-mounts `/var/lib/hyperhive/` at `/run/hive-state/`; nginx
|
||||
includes `/run/hive-state/agents.conf`. A systemd path unit
|
||||
(`hive-gateway-agents-conf.path`) inside the container watches the
|
||||
file and fires `nginx -s reload` on every atomic rename from c0re
|
||||
— no `nixos-rebuild` needed (#869).
|
||||
|
||||
c0re re-fires `agent_sockets::write` every 10s so newly-bound
|
||||
markers get picked up without needing a container-start hook in
|
||||
every lifecycle path. `write()` is idempotent: steady-state cost is
|
||||
one stat per agent per tick.
|
||||
c0re regenerates `agents.conf` (and fires the path unit → reload) on
|
||||
two triggers: every topology change (new/removed agents) and every
|
||||
10s marker poll tick (`agent_sockets::spawn_poll`). `write()` is
|
||||
idempotent — skips the rename when content is unchanged so the path
|
||||
unit doesn't fire spuriously.
|
||||
|
||||
Transition: agents that haven't flipped `useUnixSocket = true` still
|
||||
appear in `agent-ports.json` (the legacy TCP map) and the gateway
|
||||
falls back to TCP for them. A future cleanup will drop the TCP map +
|
||||
the harness's TCP bind once every agent's flipped.
|
||||
Transition: agents that haven't flipped `useUnixSocket = true` get a
|
||||
TCP loopback upstream in `agents.conf` (deterministic port from
|
||||
`agent_web_port(name)`). A future cleanup will drop the TCP fallback
|
||||
once every agent's flipped.
|
||||
|
||||
## Dashboard link shape (gateway vs direct)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,9 +1,11 @@
|
|||
//! `/var/lib/hyperhive/agent-ports.json` writer. Legacy TCP map for
|
||||
//! per-agent `/agent/<name>/` routing — the unix-socket replacement
|
||||
//! lives in `agent_sockets.rs`. The gateway reads this JSON at
|
||||
//! request-handling time rather than at gateway build time, so a
|
||||
//! `nixos-container update` of the gateway isn't needed every time
|
||||
//! an agent spawns / moves / destroys.
|
||||
//! `/var/lib/hyperhive/agent-ports.json` writer. Port map for
|
||||
//! per-agent `/agent/<name>/` TCP routing. Written alongside
|
||||
//! `agents.conf` (see `gateway_nginx.rs`) on every topology change;
|
||||
//! `gateway_nginx::render` reads it indirectly via
|
||||
//! `lifecycle::agent_web_port` to populate TCP upstreams for agents
|
||||
//! that haven't opted in to unix-socket mode yet. Also kept as a
|
||||
//! human-readable audit file — `cat agent-ports.json` shows every
|
||||
//! registered sub-agent and its deterministic port assignment.
|
||||
//!
|
||||
//! Shape (flat object keyed by logical agent name → web port):
|
||||
//!
|
||||
|
|
|
|||
|
|
@ -181,6 +181,14 @@ pub fn spawn_poll() {
|
|||
if let Err(e) = write(&names) {
|
||||
tracing::debug!(error = ?e, "agent_sockets poll write failed");
|
||||
}
|
||||
// Regenerate the gateway nginx include whenever
|
||||
// socket readiness changes — the upstream
|
||||
// selection (UDS vs TCP) depends on .bound markers
|
||||
// which change independently of topology. Write is
|
||||
// idempotent; skips rename when nothing changed.
|
||||
if let Err(e) = crate::gateway_nginx::write(&names) {
|
||||
tracing::debug!(error = ?e, "gateway_nginx poll write failed");
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::debug!(error = ?e, "agent_sockets poll: failed to list agents");
|
||||
|
|
|
|||
205
hive-c0re/src/gateway_nginx.rs
Normal file
205
hive-c0re/src/gateway_nginx.rs
Normal file
|
|
@ -0,0 +1,205 @@
|
|||
//! Runtime nginx include-file generator for the gateway's per-agent
|
||||
//! `/agent/<name>/` location blocks (#869).
|
||||
//!
|
||||
//! Writes `/var/lib/hyperhive/agents.conf` on every topology change.
|
||||
//! The gateway container bind-mounts the whole `/var/lib/hyperhive/`
|
||||
//! directory at `/run/hive-state/` and nginx includes
|
||||
//! `/run/hive-state/agents.conf`. A systemd path unit inside the
|
||||
//! gateway container watches the file and triggers `nginx -s reload`
|
||||
//! on every atomic rename — no `nixos-rebuild switch` needed when
|
||||
//! agents start, stop, or flip `useUnixSocket`.
|
||||
//!
|
||||
//! Upstream selection mirrors `agent_sockets::build_map`: an agent
|
||||
//! gets a UDS upstream when its `.bound` marker exists (harness has
|
||||
//! bound the unix socket); otherwise falls back to the deterministic
|
||||
//! TCP port from `lifecycle::agent_web_port`. Proxy headers are
|
||||
//! emitted in full so the generated file is self-contained nginx
|
||||
//! config — no dependency on which `recommendedProxySettings` knobs
|
||||
//! the host config has on.
|
||||
//!
|
||||
//! `write()` is idempotent: if the rendered body equals what's already
|
||||
//! on disk, the rename is skipped and the path unit doesn't fire.
|
||||
//! Same atomic `<path>.tmp` + `rename()` shape as `agent_ports` /
|
||||
//! `agent_sockets` — a crashing c0re process never leaves a partial
|
||||
//! file the gateway's nginx would fail to parse.
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use std::fmt::Write as _;
|
||||
use std::path::PathBuf;
|
||||
|
||||
use crate::agent_sockets;
|
||||
use crate::lifecycle::{self, MANAGER_NAME};
|
||||
|
||||
const HOST_CONF_PATH: &str = "/var/lib/hyperhive/agents.conf";
|
||||
|
||||
/// Host-side path where c0re writes the generated nginx include file.
|
||||
/// The gateway container bind-mounts `/var/lib/hyperhive/` at
|
||||
/// `/run/hive-state/` so nginx inside can read it at
|
||||
/// `/run/hive-state/agents.conf`.
|
||||
#[must_use]
|
||||
pub fn host_conf_path() -> PathBuf {
|
||||
PathBuf::from(HOST_CONF_PATH)
|
||||
}
|
||||
|
||||
/// Nginx proxy headers present in every per-agent location block.
|
||||
/// `$connection_upgrade` is defined in the http context by the NixOS
|
||||
/// nginx module when `recommendedProxySettings = true` (which the
|
||||
/// gateway always sets). `X-Forwarded-Prefix` is per-location (set
|
||||
/// below in the render loop). Kept as a standalone const so it's
|
||||
/// visible to tests without repeating the text.
|
||||
const PROXY_HEADER_BLOCK: &str = " proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection $connection_upgrade;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;";
|
||||
|
||||
/// Render the nginx include body for `names`. Manager is filtered out.
|
||||
/// Each remaining agent gets a `location /agent/<name>/` block whose
|
||||
/// `proxy_pass` targets the unix socket when the `.bound` marker
|
||||
/// exists, or TCP loopback otherwise.
|
||||
///
|
||||
/// Output is deterministic for the same (sorted) set of names +
|
||||
/// `.bound` state: no timestamps, no UUIDs. `BTreeMap` would give
|
||||
/// alphabetical order; we rely on the caller to pass sorted names if
|
||||
/// they care about diff stability, but the gateway treats the blocks
|
||||
/// as unordered by nginx's longest-prefix-match rules so ordering only
|
||||
/// affects human readability.
|
||||
fn render(names: &[String]) -> String {
|
||||
let mut out = String::from(
|
||||
"# Generated by hive-c0re \u{2014} do not edit.\
|
||||
\n# Refreshed on every topology change + when agents bind/drop their unix sockets.\
|
||||
\n# Gateway reloads nginx automatically on each update (systemd path unit).\n",
|
||||
);
|
||||
for name in names {
|
||||
if name == MANAGER_NAME {
|
||||
continue;
|
||||
}
|
||||
let port = lifecycle::agent_web_port(name);
|
||||
let upstream = if agent_sockets::ready_marker_for(name).exists() {
|
||||
format!(
|
||||
"http://unix:{}:/",
|
||||
agent_sockets::socket_path_for(name).display()
|
||||
)
|
||||
} else {
|
||||
format!("http://127.0.0.1:{port}/")
|
||||
};
|
||||
let _ = write!(
|
||||
out,
|
||||
"\nlocation /agent/{name}/ {{\n\
|
||||
{PROXY_HEADER_BLOCK}\n\
|
||||
\n proxy_pass {upstream};\n\
|
||||
\n proxy_set_header X-Forwarded-Prefix /agent/{name};\n\
|
||||
\n proxy_buffering off;\n\
|
||||
\n proxy_read_timeout 1d;\n\
|
||||
\n proxy_intercept_errors on;\n\
|
||||
\n error_page 502 503 504 = /__hive_agent_unreachable;\n\
|
||||
}}\n",
|
||||
);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Atomically write the nginx include file for `names` to
|
||||
/// [`host_conf_path()`]. Skips the rename when the rendered body
|
||||
/// matches what's already on disk (idempotent; avoids spurious
|
||||
/// gateway reloads on a quiet tick). On a fresh install where the
|
||||
/// file doesn't exist yet, writes an empty-but-valid config so nginx
|
||||
/// can start before any agents have registered.
|
||||
pub fn write(names: &[String]) -> Result<()> {
|
||||
let body = render(names);
|
||||
let path = host_conf_path();
|
||||
if std::fs::read_to_string(&path).ok().as_deref() == Some(&body) {
|
||||
return Ok(());
|
||||
}
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent)
|
||||
.with_context(|| format!("create dir {}", parent.display()))?;
|
||||
}
|
||||
let tmp = path.with_extension("conf.tmp");
|
||||
std::fs::write(&tmp, &body)
|
||||
.with_context(|| format!("write tmp {}", tmp.display()))?;
|
||||
std::fs::rename(&tmp, &path).with_context(|| {
|
||||
format!(
|
||||
"rename {} -> {} (atomic publish)",
|
||||
tmp.display(),
|
||||
path.display()
|
||||
)
|
||||
})?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn render_empty_is_header_only() {
|
||||
let body = render(&[]);
|
||||
assert!(body.starts_with("# Generated by hive-c0re"));
|
||||
// No location blocks when no agents.
|
||||
assert!(!body.contains("location"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn render_filters_manager() {
|
||||
let names: Vec<String> = [MANAGER_NAME, "iris"]
|
||||
.iter()
|
||||
.map(|s| (*s).to_owned())
|
||||
.collect();
|
||||
// Use build_map_with so we don't need real .bound files.
|
||||
// For render, we call it directly with names.
|
||||
let body = render(&names);
|
||||
assert!(!body.contains(&format!("/agent/{MANAGER_NAME}/")));
|
||||
assert!(body.contains("/agent/iris/"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn render_tcp_upstream_when_no_bound_marker() {
|
||||
// No .bound file on disk → falls back to TCP loopback.
|
||||
let names = vec!["iris".to_owned()];
|
||||
let body = render(&names);
|
||||
let port = lifecycle::agent_web_port("iris");
|
||||
assert!(
|
||||
body.contains(&format!("proxy_pass http://127.0.0.1:{port}/")),
|
||||
"expected TCP upstream for iris, got:\n{body}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn render_includes_proxy_headers() {
|
||||
let names = vec!["iris".to_owned()];
|
||||
let body = render(&names);
|
||||
// All required proxy headers present (WebSocket upgrade + forwarded-for).
|
||||
assert!(body.contains("proxy_http_version 1.1"));
|
||||
assert!(body.contains("proxy_set_header Upgrade $http_upgrade"));
|
||||
assert!(body.contains("proxy_set_header X-Forwarded-For"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn render_includes_forwarded_prefix() {
|
||||
let names = vec!["atlas".to_owned()];
|
||||
let body = render(&names);
|
||||
assert!(
|
||||
body.contains("X-Forwarded-Prefix /agent/atlas"),
|
||||
"expected X-Forwarded-Prefix for atlas, got:\n{body}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn render_includes_error_pages() {
|
||||
let names = vec!["iris".to_owned()];
|
||||
let body = render(&names);
|
||||
assert!(body.contains("proxy_intercept_errors on"));
|
||||
assert!(body.contains("__hive_agent_unreachable"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn render_includes_buffering_and_timeout() {
|
||||
let names = vec!["iris".to_owned()];
|
||||
let body = render(&names);
|
||||
assert!(body.contains("proxy_buffering off"));
|
||||
assert!(body.contains("proxy_read_timeout 1d"));
|
||||
}
|
||||
}
|
||||
|
|
@ -16,6 +16,7 @@ pub mod actions;
|
|||
pub mod agent_ports;
|
||||
pub mod agent_server;
|
||||
pub mod agent_sockets;
|
||||
pub mod gateway_nginx;
|
||||
pub mod approvals;
|
||||
pub mod auto_update;
|
||||
pub mod broker;
|
||||
|
|
|
|||
|
|
@ -126,6 +126,16 @@ pub async fn sync_agents(
|
|||
tracing::warn!(error = ?e, "agent_sockets::write failed (non-fatal)");
|
||||
}
|
||||
|
||||
// Refresh /var/lib/hyperhive/agents.conf — the nginx include file
|
||||
// the gateway picks up at runtime without needing a
|
||||
// nixos-rebuild. The gateway container bind-mounts
|
||||
// /var/lib/hyperhive/ and a systemd path unit fires
|
||||
// `nginx -s reload` when this file changes (#869). Same
|
||||
// best-effort + non-fatal shape.
|
||||
if let Err(e) = crate::gateway_nginx::write(&agent_names) {
|
||||
tracing::warn!(error = ?e, "gateway_nginx::write failed (non-fatal)");
|
||||
}
|
||||
|
||||
if initial {
|
||||
git(&dir, &["init", "--initial-branch=main"]).await?;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,43 +11,6 @@ let
|
|||
forgeCfg = config.services.hyperhive.forge;
|
||||
networkCfg = config.services.hyperhive.network;
|
||||
|
||||
# Per-agent port table for `/agent/<name>/` routing. C0re writes
|
||||
# this JSON on every topology change; gateway reads at deploy time.
|
||||
# Missing file → empty map → no per-agent routes (graceful default).
|
||||
# See `docs/gateway.md` for the discovery + rebuild flow.
|
||||
agentPortsTable =
|
||||
if cfg.agentPortsFile == null || !builtins.pathExists cfg.agentPortsFile then
|
||||
{ }
|
||||
else
|
||||
builtins.fromJSON (builtins.readFile cfg.agentPortsFile);
|
||||
|
||||
# Per-agent unix-socket table for `/agent/<name>/` UDS upstream
|
||||
# (#784 phase 2 step 3). C0re writes this JSON alongside
|
||||
# agent-ports.json; gateway reads at deploy time. Per-agent the
|
||||
# entry wins over the TCP port. Missing entry (or missing file)
|
||||
# → fall back to the TCP port. See
|
||||
# `docs/gateway.md::Per-agent UDS upstream (#784)`.
|
||||
agentSocketsTable =
|
||||
if cfg.agentSocketsFile == null || !builtins.pathExists cfg.agentSocketsFile then
|
||||
{ }
|
||||
else
|
||||
builtins.fromJSON (builtins.readFile cfg.agentSocketsFile);
|
||||
|
||||
# Resolve a per-agent upstream URL. Socket entry wins ONLY when the
|
||||
# socket file actually exists at eval time — guards against agents
|
||||
# that have an `agent-sockets.json` entry from c0re's blanket emit
|
||||
# but haven't actually flipped `hyperhive.web.useUnixSocket = true`
|
||||
# (their harness still binds TCP only, so a UDS upstream would 502).
|
||||
# Falls back to the TCP loopback otherwise. Once c0re ships the
|
||||
# `.bound` marker filter (#784 step 2d follow-up), the path-exists
|
||||
# check becomes redundant but harmless; step 4 drops it entirely.
|
||||
agentUpstreamFor =
|
||||
name: port:
|
||||
if agentSocketsTable ? ${name} && builtins.pathExists agentSocketsTable.${name} then
|
||||
"http://unix:${agentSocketsTable.${name}}:/"
|
||||
else
|
||||
"http://127.0.0.1:${toString port}/";
|
||||
|
||||
# Static error pages for `/agent/<name>/` mishaps (#755). Mara's
|
||||
# call: useful pages instead of nginx's default 404/502 for routes
|
||||
# we've already special-cased. See `docs/gateway.md::Per-agent
|
||||
|
|
@ -71,7 +34,7 @@ let
|
|||
<body>
|
||||
<h1>◆ agent not found</h1>
|
||||
<p>No agent matches the requested <code>/agent/<name>/</code> path on this hive.</p>
|
||||
<p>Operator: check the agent name in <a href="/">the dashboard</a> — the gateway picks up new agents on the next <code>nixos-rebuild switch</code>.</p>
|
||||
<p>Operator: check the agent name in <a href="/">the dashboard</a>.</p>
|
||||
</body>
|
||||
</html>
|
||||
EOF
|
||||
|
|
@ -237,75 +200,6 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
agentPortsFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.path;
|
||||
default = "/var/lib/hyperhive/agent-ports.json";
|
||||
example = "/var/lib/hyperhive/agent-ports.json";
|
||||
description = ''
|
||||
Path to a JSON file mapping sub-agent names to their web ports
|
||||
for `/agent/<name>/` routing through the gateway (#15 v0).
|
||||
Shape: `{ "<name>": <port>, ... }`. Written by hive-c0re on
|
||||
every topology change (the rust side knows the canonical port
|
||||
allocation via `lifecycle::agent_web_port`; the gateway just
|
||||
reads what it's told).
|
||||
|
||||
For each `<name>: <port>` entry, the gateway adds a
|
||||
`location /agent/<name>/` block that `proxy_pass`es to
|
||||
`http://127.0.0.1:<port>/`. Empty / missing file → no
|
||||
per-agent routes generated → gateway falls back to its pre-#15
|
||||
shape (just `/` + matrix surfaces).
|
||||
|
||||
**Purely additive**: the old `http://<host>:<port>/` direct
|
||||
reach keeps working in parallel; this just gives the operator
|
||||
a single-origin route. Manager isn't included in the map (no
|
||||
per-agent prefix needed; manager already gets the `/` route
|
||||
via the c0re upstream block).
|
||||
|
||||
Set to `null` to disable per-agent routing entirely without
|
||||
creating the file. Set to a custom path if the operator's c0re
|
||||
writes the table elsewhere.
|
||||
|
||||
**Rebuild trigger**: the gateway container picks up new entries
|
||||
on the next `nixos-rebuild switch` (or `hivectl gateway-sync`
|
||||
if that helper lands). c0re writes are not auto-applied to a
|
||||
running gateway — see the follow-up in #15 for runtime nginx
|
||||
include + reload + eventual per-agent unix sockets.
|
||||
'';
|
||||
};
|
||||
|
||||
agentSocketsFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.path;
|
||||
default = "/var/lib/hyperhive/agent-sockets.json";
|
||||
example = "/var/lib/hyperhive/agent-sockets.json";
|
||||
description = ''
|
||||
Path to a JSON file mapping sub-agent names to their
|
||||
per-agent unix-socket paths for `/agent/<name>/` UDS upstream
|
||||
routing (#784 phase 2 step 3). Shape:
|
||||
`{ "<name>": "/run/hive-agent/<name>/web.sock", ... }`.
|
||||
Written by hive-c0re alongside `agentPortsFile` on every
|
||||
topology change (`hive_c0re::agent_sockets::write`;
|
||||
path-shape derives from
|
||||
`agent_sockets::socket_path_for(name)`).
|
||||
|
||||
Per-agent, the socket entry wins over the TCP port: when an
|
||||
agent appears in this map, the gateway's `proxy_pass` for
|
||||
that agent's `/agent/<name>/` location targets
|
||||
`http://unix:<path>:/` instead of `http://127.0.0.1:<port>/`.
|
||||
Agents that haven't opted in (no `HIVE_WEB_SOCKET` set,
|
||||
no entry in the JSON, or both files unset) fall back to
|
||||
TCP via `agentPortsFile`. Coexists with the TCP map during
|
||||
the rollout — eventually drops `agentPortsFile` entirely
|
||||
when every agent's flipped (#784 step 4).
|
||||
|
||||
Set to `null` to skip UDS upstreams entirely (gateway uses
|
||||
TCP for every agent regardless of what hive-c0re writes).
|
||||
|
||||
**Bind-mount requirement**: when this is enabled the gateway
|
||||
container needs `/run/hive-agent/` bind-mounted from the
|
||||
host. Handled automatically by `containers.hive-gateway`
|
||||
below when at least one socket entry exists.
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
config = lib.mkIf cfg.enable {
|
||||
|
|
@ -320,14 +214,21 @@ in
|
|||
}
|
||||
];
|
||||
|
||||
# Ensure the per-agent UDS bind-mount source exists at host boot,
|
||||
# before the gateway container's first start. nspawn would
|
||||
# auto-create an empty dir if missing (argus 🟡 on #829), but a
|
||||
# tmpfiles rule makes the intent explicit and dodges the
|
||||
# fresh-boot-before-any-agent-spawn window where the dir wouldn't
|
||||
# exist yet from c0re's per-agent `set_nspawn_flags` mkdir chain.
|
||||
# Ensure bind-mount sources exist at host boot before the gateway
|
||||
# container's first start. nspawn would auto-create missing dirs
|
||||
# (argus 🟡 on #829), but tmpfiles rules make the intent explicit
|
||||
# and cover the fresh-boot window before c0re has run.
|
||||
#
|
||||
# /run/hive-agent — per-agent UDS socket dir, written by c0re's
|
||||
# set_nspawn_flags when agents start.
|
||||
# /var/lib/hyperhive — hyperhive state dir, created by c0re on
|
||||
# first run. Also pre-seed agents.conf with an empty-but-valid
|
||||
# header so nginx can start + include the file before c0re writes
|
||||
# its first real content (f = create-if-absent, no overwrite).
|
||||
systemd.tmpfiles.rules = [
|
||||
"d /run/hive-agent 0755 root root - -"
|
||||
"d /var/lib/hyperhive 0755 root root - -"
|
||||
"f /var/lib/hyperhive/agents.conf 0644 root root - # Generated by hive-c0re — do not edit.\n"
|
||||
];
|
||||
|
||||
containers.hive-gateway = {
|
||||
|
|
@ -339,16 +240,25 @@ in
|
|||
# layer that matters.
|
||||
privateNetwork = false;
|
||||
# Bind-mount the per-agent socket dir so nginx inside the gateway
|
||||
# container can `connect(2)` to the UDS upstreams hive-c0re
|
||||
# publishes in `agent-sockets.json` (#784 phase 2 step 3).
|
||||
# Read-only (we don't bind anything here; just connect). Mount
|
||||
# is unconditional but inert when no agents have opted in:
|
||||
# agent-sockets.json missing/empty → `agentSocketsTable = {}`
|
||||
# → every per-agent location uses the TCP fallback.
|
||||
# container can `connect(2)` to the UDS upstreams (#784 step 3).
|
||||
# Read-only (we just connect; harness writes the socket inside
|
||||
# the agent's own container). Host-side dir is pre-created by a
|
||||
# tmpfiles rule so nspawn always finds a source at boot.
|
||||
bindMounts."/run/hive-agent" = {
|
||||
hostPath = "/run/hive-agent";
|
||||
isReadOnly = true;
|
||||
};
|
||||
# Bind-mount the hyperhive state dir so nginx can include the
|
||||
# runtime-generated agents.conf. Read-only; c0re writes
|
||||
# /var/lib/hyperhive/agents.conf on the host and the systemd
|
||||
# path unit inside the container triggers nginx -s reload on
|
||||
# each atomic rename (#869). Pre-created by a tmpfiles rule so
|
||||
# nspawn always finds the source at boot (c0re also writes it
|
||||
# on first startup, but the container may start before c0re).
|
||||
bindMounts."/run/hive-state" = {
|
||||
hostPath = "/var/lib/hyperhive";
|
||||
isReadOnly = true;
|
||||
};
|
||||
config =
|
||||
{ pkgs, ... }:
|
||||
let
|
||||
|
|
@ -397,8 +307,7 @@ in
|
|||
publicScheme = if cfg.selfSignedTls then "https" else "http";
|
||||
publicPort = if cfg.selfSignedTls then cfg.httpsPort else cfg.port;
|
||||
publicPortDefault = if cfg.selfSignedTls then 443 else 80;
|
||||
publicPortSuffix =
|
||||
if publicPort == publicPortDefault then "" else ":${toString publicPort}";
|
||||
publicPortSuffix = if publicPort == publicPortDefault then "" else ":${toString publicPort}";
|
||||
in
|
||||
{
|
||||
system.stateVersion = "26.05";
|
||||
|
|
@ -479,6 +388,38 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
# Watch /run/hive-state/agents.conf (bind-mounted from the
|
||||
# host's /var/lib/hyperhive/agents.conf) for changes and
|
||||
# trigger an nginx reload when c0re atomically renames a new
|
||||
# version into place (#869). PathChanged fires on
|
||||
# IN_CLOSE_WRITE + IN_MOVED_TO, so the atomic rename c0re
|
||||
# uses (write .conf.tmp → rename) wakes the path unit.
|
||||
# The reload is a no-op if the new config is identical —
|
||||
# gateway_nginx::write skips the rename when content is
|
||||
# unchanged, so the path unit doesn't fire at all on quiet
|
||||
# ticks.
|
||||
systemd.paths.hive-gateway-agents-conf = {
|
||||
wantedBy = [ "nginx.service" ];
|
||||
after = [ "nginx.service" ];
|
||||
pathConfig = {
|
||||
PathChanged = "/run/hive-state/agents.conf";
|
||||
Unit = "hive-gateway-nginx-reload.service";
|
||||
};
|
||||
};
|
||||
|
||||
systemd.services.hive-gateway-nginx-reload = {
|
||||
description = "Reload nginx after agents.conf change (#869)";
|
||||
# Don't block any target — fires only when the path unit
|
||||
# triggers it.
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
# nginx -s reload sends SIGHUP to the master process via
|
||||
# the pid file. Runs as root inside the container (pid 1
|
||||
# is the nspawn init; nginx master starts as root).
|
||||
ExecStart = "/run/current-system/sw/bin/nginx -s reload";
|
||||
};
|
||||
};
|
||||
|
||||
services.nginx = {
|
||||
enable = true;
|
||||
recommendedProxySettings = true;
|
||||
|
|
@ -551,38 +492,17 @@ in
|
|||
};
|
||||
}
|
||||
)
|
||||
//
|
||||
# Per-agent UIs (#15 v0; UDS upstream #784 step 3).
|
||||
# One `/agent/<name>/` block per entry in
|
||||
# `agentPortsTable`. `agentUpstreamFor` resolves
|
||||
# to `http://unix:<path>:/` when the agent has
|
||||
# opted in via `hyperhive.web.useUnixSocket` (and
|
||||
# appears in `agentSocketsTable`); otherwise
|
||||
# `http://127.0.0.1:<port>/`. Trailing-slash pair
|
||||
# strips the prefix; `X-Forwarded-Prefix` lets the
|
||||
# harness build absolute URLs when relative isn't
|
||||
# enough. `proxy_intercept_errors` + `error_page` rewrite
|
||||
# upstream 502/503/504 to `unreachable.html` (#755).
|
||||
lib.mapAttrs' (name: port: {
|
||||
name = "/agent/${name}/";
|
||||
value = {
|
||||
proxyPass = agentUpstreamFor name port;
|
||||
proxyWebsockets = true;
|
||||
extraConfig = ''
|
||||
proxy_set_header X-Forwarded-Prefix /agent/${name};
|
||||
proxy_buffering off;
|
||||
proxy_read_timeout 1d;
|
||||
proxy_intercept_errors on;
|
||||
error_page 502 503 504 = /__hive_agent_unreachable;
|
||||
'';
|
||||
};
|
||||
}) agentPortsTable
|
||||
//
|
||||
# `/agent/` catch-all (#755): hits when an operator
|
||||
# requests `/agent/<unknown>/...` — a name not in
|
||||
# `agentPortsTable`. Without this it falls through to
|
||||
# `/` (c0re dashboard upstream) which returns 404
|
||||
# with no useful context. Custom 404 page instead.
|
||||
# requests `/agent/<unknown>/...`. Without this the
|
||||
# request falls through to `/` (c0re dashboard) and
|
||||
# returns 404 with no useful context. Custom 404
|
||||
# page instead. Per-agent `location /agent/<name>/`
|
||||
# blocks live in `/run/hive-state/agents.conf` —
|
||||
# nginx picks them up via the `include` in
|
||||
# `extraConfig` below; the catch-all only matches
|
||||
# names that aren't in that file (nginx longest-
|
||||
# prefix-match: `/agent/atlas/` beats `/agent/`).
|
||||
{
|
||||
"/agent/" = {
|
||||
extraConfig = ''
|
||||
|
|
@ -624,6 +544,18 @@ in
|
|||
'';
|
||||
};
|
||||
};
|
||||
# Per-agent location blocks, generated at runtime by
|
||||
# hive-c0re and written to /var/lib/hyperhive/agents.conf
|
||||
# on the host. The bind-mount at /run/hive-state/ exposes
|
||||
# that file here. nginx parses `include` at config-load
|
||||
# time so a reload (triggered by the hive-gateway-nginx-
|
||||
# reload path unit when agents.conf changes) picks up new
|
||||
# or removed agents without a nixos-rebuild. nginx's
|
||||
# longest-prefix-match rule ensures `/agent/<name>/` from
|
||||
# this file beats the `/agent/` catch-all above (#869).
|
||||
extraConfig = ''
|
||||
include /run/hive-state/agents.conf;
|
||||
'';
|
||||
};
|
||||
}
|
||||
//
|
||||
|
|
|
|||
Loading…
Reference in a new issue