gateway: send nginx stub_status to VictoriaMetrics, add a gateway dashboard
The gateway nginx serves stub_status on a loopback server of its own, 127.0.0.1:8085 (services.hyperhive.gateway.statusPort). The hive-tier collector reads it with the contrib nginx receiver, which the pinned otelcol-contrib 0.151.0 ships, so no exporter daemon is added. The receiver dials http:// only (a unix: or http+unix: endpoint fails with "unsupported protocol scheme"), so the listener is loopback TCP with allow 127.0.0.1 / deny all rather than a unix socket. The new gateway dashboard plots nginx.requests, nginx.connections_current by state, and nginx.connections_accepted against handled, per hive. Part of #5342.
This commit is contained in:
parent
e59a2a34fb
commit
f70bbda7d7
8 changed files with 460 additions and 1 deletions
|
|
@ -129,6 +129,8 @@ nginx is the one external entry point. The per-agent web-port range (`8100`–`8
|
|||
|
||||
The dashboard port (`services.hyperhive.c0re.dashboardPort`, default 7000) binds `127.0.0.1` only, so remote dashboard access goes through the gateway. The dashboard can stop, rebuild and destroy agents; don't expose it without a reverse proxy in front.
|
||||
|
||||
nginx serves its `stub_status` counters on `127.0.0.1:<statusPort>` (`services.hyperhive.gateway.statusPort`, default 8085) at `/stub_status`, from a server of its own outside the vhost tree. The hive's collector reads them with its `nginx` receiver, and the `gateway` Grafana dashboard plots them. The receiver only dials `http://`, which is why this is loopback TCP rather than a unix socket.
|
||||
|
||||
## `HIVE_FORGE_URL`: agents reach the forge via the gateway by domain
|
||||
|
||||
Agents poll `HIVE_FORGE_URL` for Forgejo notifications and run every `hive-forge` call against it. `nix/host-modules/hive-c0re/environment.nix` sets it to `http://<swarm.forge.domain>` (default `forge.<swarm-domain>` — a swarm runs one forge). Agents run in a private netns and can't reach the host's loopback, so they resolve that name through the bridge dnsmasq to the bridge IP and reach nginx on port 80 (the bridge firewall opens 80 and 443) — the same path an operator's browser takes.
|
||||
|
|
|
|||
|
|
@ -176,6 +176,7 @@ kept.
|
|||
- `claude-usage`: Claude use by model, effort, token type and source.
|
||||
- `controller`: the swarm controller's job graph by state, and the push and commit rates it sees on the forge.
|
||||
- `forge`: what the forge holds, and the forge process's CPU, memory and descriptors.
|
||||
- `gateway`: the gateway nginx on each hive's host: requests per second, connections by state, and accepted against handled connections.
|
||||
- `collector`: what the OTEL collectors accept, refuse, export and queue, with queue depth against capacity.
|
||||
- `logstore`, `metricstore`: health of the log store and the metrics store.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue