vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

gateway: hot-reload agents.conf at runtime (#869)

Browse source 
Replace eval-time per-agent nginx location baking with a runtime
include file. c0re writes /var/lib/hyperhive/agents.conf (nginx
location blocks, UDS or TCP per agent) on every topology change and
on the 10s marker poll. The gateway container bind-mounts
/var/lib/hyperhive/ at /run/hive-state/ and nginx includes
/run/hive-state/agents.conf. A systemd path unit inside the container
watches the file for changes and fires `nginx -s reload` on each
atomic rename from c0re — no nixos-rebuild switch needed when agents
start, stop, or flip useUnixSocket.

  - new hive-c0re/src/gateway_nginx.rs: write() + render()
  - lib.rs + meta.rs + agent_sockets::spawn_poll: hook in write()
  - hive-gateway.nix: drop agentPortsTable/agentSocketsTable/
    agentUpstreamFor/lib.mapAttrs', add /run/hive-state bind-mount,
    include directive, systemd path unit + reload service, tmpfiles
    for /var/lib/hyperhive + agents.conf seed
  - docs/gateway.md: update vhost table + Per-agent UDS section
This commit is contained in:
atlas 2026-05-31 20:12:00 +02:00 • committed by mara
commit 07434e8f50
7 changed files with 337 additions and 173 deletions
Download patch file Download diff file Expand all files Collapse all files

234
nix/modules/hive-gateway.nix
View file

@ -11,43 +11,6 @@ let
forgeCfg = config.services.hyperhive.forge;
networkCfg = config.services.hyperhive.network;
# Per-agent port table for `/agent/<name>/` routing. C0re writes
# this JSON on every topology change; gateway reads at deploy time.
# Missing file → empty map → no per-agent routes (graceful default).
# See `docs/gateway.md` for the discovery + rebuild flow.
agentPortsTable =
if cfg.agentPortsFile == null || !builtins.pathExists cfg.agentPortsFile then
{ }
else
builtins.fromJSON (builtins.readFile cfg.agentPortsFile);
# Per-agent unix-socket table for `/agent/<name>/` UDS upstream
# (#784 phase 2 step 3). C0re writes this JSON alongside
# agent-ports.json; gateway reads at deploy time. Per-agent the
# entry wins over the TCP port. Missing entry (or missing file)
# → fall back to the TCP port. See
# `docs/gateway.md::Per-agent UDS upstream (#784)`.
agentSocketsTable =
if cfg.agentSocketsFile == null || !builtins.pathExists cfg.agentSocketsFile then
{ }
else
builtins.fromJSON (builtins.readFile cfg.agentSocketsFile);
# Resolve a per-agent upstream URL. Socket entry wins ONLY when the
# socket file actually exists at eval time — guards against agents
# that have an `agent-sockets.json` entry from c0re's blanket emit
# but haven't actually flipped `hyperhive.web.useUnixSocket = true`
# (their harness still binds TCP only, so a UDS upstream would 502).
# Falls back to the TCP loopback otherwise. Once c0re ships the
# `.bound` marker filter (#784 step 2d follow-up), the path-exists
# check becomes redundant but harmless; step 4 drops it entirely.
agentUpstreamFor =
name: port:
if agentSocketsTable ? ${name} && builtins.pathExists agentSocketsTable.${name} then
"http://unix:${agentSocketsTable.${name}}:/"
else
"http://127.0.0.1:${toString port}/";
# Static error pages for `/agent/<name>/` mishaps (#755). Mara's
# call: useful pages instead of nginx's default 404/502 for routes
# we've already special-cased. See `docs/gateway.md::Per-agent
@ -71,7 +34,7 @@ let
<body>
<h1>◆ agent not found</h1>
<p>No agent matches the requested <code>/agent/&lt;name&gt;/</code> path on this hive.</p>
<p>Operator: check the agent name in <a href="/">the dashboard</a> — the gateway picks up new agents on the next <code>nixos-rebuild switch</code>.</p>
<p>Operator: check the agent name in <a href="/">the dashboard</a>.</p>
</body>
</html>
EOF
@ -237,75 +200,6 @@ in
'';
};
agentPortsFile = lib.mkOption {
type = lib.types.nullOr lib.types.path;
default = "/var/lib/hyperhive/agent-ports.json";
example = "/var/lib/hyperhive/agent-ports.json";
description = ''
Path to a JSON file mapping sub-agent names to their web ports
for `/agent/<name>/` routing through the gateway (#15 v0).
Shape: `{ "<name>": <port>, ... }`. Written by hive-c0re on
every topology change (the rust side knows the canonical port
allocation via `lifecycle::agent_web_port`; the gateway just
reads what it's told).
For each `<name>: <port>` entry, the gateway adds a
`location /agent/<name>/` block that `proxy_pass`es to
`http://127.0.0.1:<port>/`. Empty / missing file → no
per-agent routes generated → gateway falls back to its pre-#15
shape (just `/` + matrix surfaces).
**Purely additive**: the old `http://<host>:<port>/` direct
reach keeps working in parallel; this just gives the operator
a single-origin route. Manager isn't included in the map (no
per-agent prefix needed; manager already gets the `/` route
via the c0re upstream block).
Set to `null` to disable per-agent routing entirely without
creating the file. Set to a custom path if the operator's c0re
writes the table elsewhere.
**Rebuild trigger**: the gateway container picks up new entries
on the next `nixos-rebuild switch` (or `hivectl gateway-sync`
if that helper lands). c0re writes are not auto-applied to a
running gateway — see the follow-up in #15 for runtime nginx
include + reload + eventual per-agent unix sockets.
'';
};
agentSocketsFile = lib.mkOption {
type = lib.types.nullOr lib.types.path;
default = "/var/lib/hyperhive/agent-sockets.json";
example = "/var/lib/hyperhive/agent-sockets.json";
description = ''
Path to a JSON file mapping sub-agent names to their
per-agent unix-socket paths for `/agent/<name>/` UDS upstream
routing (#784 phase 2 step 3). Shape:
`{ "<name>": "/run/hive-agent/<name>/web.sock", ... }`.
Written by hive-c0re alongside `agentPortsFile` on every
topology change (`hive_c0re::agent_sockets::write`;
path-shape derives from
`agent_sockets::socket_path_for(name)`).
Per-agent, the socket entry wins over the TCP port: when an
agent appears in this map, the gateway's `proxy_pass` for
that agent's `/agent/<name>/` location targets
`http://unix:<path>:/` instead of `http://127.0.0.1:<port>/`.
Agents that haven't opted in (no `HIVE_WEB_SOCKET` set,
no entry in the JSON, or both files unset) fall back to
TCP via `agentPortsFile`. Coexists with the TCP map during
the rollout — eventually drops `agentPortsFile` entirely
when every agent's flipped (#784 step 4).
Set to `null` to skip UDS upstreams entirely (gateway uses
TCP for every agent regardless of what hive-c0re writes).
**Bind-mount requirement**: when this is enabled the gateway
container needs `/run/hive-agent/` bind-mounted from the
host. Handled automatically by `containers.hive-gateway`
below when at least one socket entry exists.
'';
};
};
config = lib.mkIf cfg.enable {
@ -320,14 +214,21 @@ in
}
];
# Ensure the per-agent UDS bind-mount source exists at host boot,
# before the gateway container's first start. nspawn would
# auto-create an empty dir if missing (argus 🟡 on #829), but a
# tmpfiles rule makes the intent explicit and dodges the
# fresh-boot-before-any-agent-spawn window where the dir wouldn't
# exist yet from c0re's per-agent `set_nspawn_flags` mkdir chain.
# Ensure bind-mount sources exist at host boot before the gateway
# container's first start. nspawn would auto-create missing dirs
# (argus 🟡 on #829), but tmpfiles rules make the intent explicit
# and cover the fresh-boot window before c0re has run.
#
# /run/hive-agent — per-agent UDS socket dir, written by c0re's
# set_nspawn_flags when agents start.
# /var/lib/hyperhive — hyperhive state dir, created by c0re on
# first run. Also pre-seed agents.conf with an empty-but-valid
# header so nginx can start + include the file before c0re writes
# its first real content (f = create-if-absent, no overwrite).
systemd.tmpfiles.rules = [
"d /run/hive-agent 0755 root root - -"
"d /var/lib/hyperhive 0755 root root - -"
"f /var/lib/hyperhive/agents.conf 0644 root root - # Generated by hive-c0re — do not edit.\n"
];
containers.hive-gateway = {
@ -339,16 +240,25 @@ in
# layer that matters.
privateNetwork = false;
# Bind-mount the per-agent socket dir so nginx inside the gateway
# container can `connect(2)` to the UDS upstreams hive-c0re
# publishes in `agent-sockets.json` (#784 phase 2 step 3).
# Read-only (we don't bind anything here; just connect). Mount
# is unconditional but inert when no agents have opted in:
# agent-sockets.json missing/empty → `agentSocketsTable = {}`
# → every per-agent location uses the TCP fallback.
# container can `connect(2)` to the UDS upstreams (#784 step 3).
# Read-only (we just connect; harness writes the socket inside
# the agent's own container). Host-side dir is pre-created by a
# tmpfiles rule so nspawn always finds a source at boot.
bindMounts."/run/hive-agent" = {
hostPath = "/run/hive-agent";
isReadOnly = true;
};
# Bind-mount the hyperhive state dir so nginx can include the
# runtime-generated agents.conf. Read-only; c0re writes
# /var/lib/hyperhive/agents.conf on the host and the systemd
# path unit inside the container triggers nginx -s reload on
# each atomic rename (#869). Pre-created by a tmpfiles rule so
# nspawn always finds the source at boot (c0re also writes it
# on first startup, but the container may start before c0re).
bindMounts."/run/hive-state" = {
hostPath = "/var/lib/hyperhive";
isReadOnly = true;
};
config =
{ pkgs, ... }:
let
@ -397,8 +307,7 @@ in
publicScheme = if cfg.selfSignedTls then "https" else "http";
publicPort = if cfg.selfSignedTls then cfg.httpsPort else cfg.port;
publicPortDefault = if cfg.selfSignedTls then 443 else 80;
publicPortSuffix =
if publicPort == publicPortDefault then "" else ":${toString publicPort}";
publicPortSuffix = if publicPort == publicPortDefault then "" else ":${toString publicPort}";
in
{
system.stateVersion = "26.05";
@ -479,6 +388,38 @@ in
'';
};
# Watch /run/hive-state/agents.conf (bind-mounted from the
# host's /var/lib/hyperhive/agents.conf) for changes and
# trigger an nginx reload when c0re atomically renames a new
# version into place (#869). PathChanged fires on
# IN_CLOSE_WRITE + IN_MOVED_TO, so the atomic rename c0re
# uses (write .conf.tmp → rename) wakes the path unit.
# The reload is a no-op if the new config is identical —
# gateway_nginx::write skips the rename when content is
# unchanged, so the path unit doesn't fire at all on quiet
# ticks.
systemd.paths.hive-gateway-agents-conf = {
wantedBy = [ "nginx.service" ];
after = [ "nginx.service" ];
pathConfig = {
PathChanged = "/run/hive-state/agents.conf";
Unit = "hive-gateway-nginx-reload.service";
};
};
systemd.services.hive-gateway-nginx-reload = {
description = "Reload nginx after agents.conf change (#869)";
# Don't block any target — fires only when the path unit
# triggers it.
serviceConfig = {
Type = "oneshot";
# nginx -s reload sends SIGHUP to the master process via
# the pid file. Runs as root inside the container (pid 1
# is the nspawn init; nginx master starts as root).
ExecStart = "/run/current-system/sw/bin/nginx -s reload";
};
};
services.nginx = {
enable = true;
recommendedProxySettings = true;
@ -551,38 +492,17 @@ in
};
}
)
//
# Per-agent UIs (#15 v0; UDS upstream #784 step 3).
# One `/agent/<name>/` block per entry in
# `agentPortsTable`. `agentUpstreamFor` resolves
# to `http://unix:<path>:/` when the agent has
# opted in via `hyperhive.web.useUnixSocket` (and
# appears in `agentSocketsTable`); otherwise
# `http://127.0.0.1:<port>/`. Trailing-slash pair
# strips the prefix; `X-Forwarded-Prefix` lets the
# harness build absolute URLs when relative isn't
# enough. `proxy_intercept_errors` + `error_page` rewrite
# upstream 502/503/504 to `unreachable.html` (#755).
lib.mapAttrs' (name: port: {
name = "/agent/${name}/";
value = {
proxyPass = agentUpstreamFor name port;
proxyWebsockets = true;
extraConfig = ''
proxy_set_header X-Forwarded-Prefix /agent/${name};
proxy_buffering off;
proxy_read_timeout 1d;
proxy_intercept_errors on;
error_page 502 503 504 = /__hive_agent_unreachable;
'';
};
}) agentPortsTable
//
# `/agent/` catch-all (#755): hits when an operator
# requests `/agent/<unknown>/...` — a name not in
# `agentPortsTable`. Without this it falls through to
# `/` (c0re dashboard upstream) which returns 404
# with no useful context. Custom 404 page instead.
# requests `/agent/<unknown>/...`. Without this the
# request falls through to `/` (c0re dashboard) and
# returns 404 with no useful context. Custom 404
# page instead. Per-agent `location /agent/<name>/`
# blocks live in `/run/hive-state/agents.conf` —
# nginx picks them up via the `include` in
# `extraConfig` below; the catch-all only matches
# names that aren't in that file (nginx longest-
# prefix-match: `/agent/atlas/` beats `/agent/`).
{
"/agent/" = {
extraConfig = ''
@ -624,6 +544,18 @@ in
'';
};
};
# Per-agent location blocks, generated at runtime by
# hive-c0re and written to /var/lib/hyperhive/agents.conf
# on the host. The bind-mount at /run/hive-state/ exposes
# that file here. nginx parses `include` at config-load
# time so a reload (triggered by the hive-gateway-nginx-
# reload path unit when agents.conf changes) picks up new
# or removed agents without a nixos-rebuild. nginx's
# longest-prefix-match rule ensures `/agent/<name>/` from
# this file beats the `/agent/` catch-all above (#869).
extraConfig = ''
include /run/hive-state/agents.conf;
'';
};
}
//