Distinguish RecvError::Lagged from Closed in the build-log stream
loop: Lagged continues (next recv delivers the full accumulated delta
via saved cursors), Closed returns (shutdown path).
Clear pre.textContent and reset cursor lengths in es.onerror when
readyState is CONNECTING so auto-reconnect doesn't double-append the
replay from cursor=0.
Backend (hive-c0re):
- build_logs.rs: add tokio::sync::broadcast::Sender<i64> to BuildLogs;
append() and finish() notify subscribers after each sqlite write.
Add BuildLogProgress struct and get_progress(id, stdout_cursor,
stderr_cursor) for incremental delta reads.
- dashboard.rs: two new endpoints —
GET /api/build-logs/id/{id}/stream SSE; streams BuildLogFrame
{stdout_append, stderr_append, status?, done} deltas until the
build finishes or the browser disconnects. Backed by an mpsc
channel task that watches the per-build broadcast notifications.
GET /api/build-logs/id/{id}/raw text/plain download with
Content-Disposition: attachment; filename build-log-{agent}-{id}.txt
Frontend (dashboard):
- tabs.js: running builds (status === null) connect an EventSource to
/stream and append lines live; "live" badge pulses amber while active,
flips to ok/fail on done. Finished builds still use the JSON fetch path.
Collapsing a running panel closes the EventSource; re-expanding
reconnects. Adds a "⬇ raw" download link to every expanded row.
- dashboard.css: .build-logs-dl inline download link; .build-logs-live
live pulse @keyframes animation.
Docs: web-ui.md updated for all three new endpoints + behaviour.
The harness runs as the non-root agent user; the per-agent
/run/hive-agent/<name>/ dir lands at 0755 root:root after
create_dir_all, so bind(2) of web.sock failed with EACCES. The error
was invisible because the web_ui::serve future was tokio::spawn'd with
its JoinHandle dropped — no log, no socket, agent looks unreachable
through the gateway.
Damocles's backend (c41bf1b) landed HYPERHIVE_HIVE_NAME / _SWARM_NAME
env vars and identity.rs accessors. This commit wires them into the two
web surfaces:
hive-c0re/dashboard.rs:
StateSnapshot gains `hive_name` + `swarm_name` (Option<String>),
populated from the env vars the c0re NixOS module injects.
hive-ag3nt/web_ui.rs:
Per-agent StateSnapshot gains the same two fields, populated from
crate::identity::hive_name() / swarm_name().
Dashboard frontend (index.html + tabs.js):
A `<p class="banner-thin" id="hive-identity">` sits above the tab
strip in the chrome. refreshState() populates it with "swarm / hive"
(or just "hive") when the fields are non-null, and updates
document.title to "<swarm>/<hive> // h1ve-c0re". No change to
chrome when both fields are null (backward compat).
Per-agent frontend (index.html + app.js + agent.css):
A hidden `.agent-hive-label` span under the title row shows the
swarm/hive label once setHeader() receives non-null names.
document.title is updated to "<label> // <hive_name>" when hive_name
is set, enabling browser-tab disambiguation when multiple hives are
open in parallel. Styled as a small uppercase purple-dim sub-label.
Every other Coordinator opener takes the broker.sqlite path and
derives its parent internally; BuildLogs alone wants a directory.
Passing the file path tripped create_dir_all on an existing file
(File exists os error 17) and blocked c0re boot.
Wires the Phase 1 sqlite store into the dashboard HTTP layer so the
frontend can surface full build logs without hitting journald.
Two new read endpoints:
- GET /api/build-logs/{agent}?limit=N — Vec<BuildLogHeader> JSON,
newest first, default limit 10, server-side cap at 50.
- GET /api/build-logs/id/{id} — BuildLogFull JSON (header +
stdout + stderr), HTTP 404 on vacuum-reaped / unknown ids.
Agent-name validated ([a-z0-9_-], 1-63 chars) before the sqlite call.
Both handlers reach coord.build_logs (Arc<BuildLogs>) introduced in
the Phase 1 commit. Docs updated in docs/web-ui.md.
- gateway_nginx.rs: HOST_CONF_PATH → /var/lib/hyperhive/gateway/agents.conf
- hive-gateway.nix: hostPath = /var/lib/hyperhive/gateway (not whole parent
dir — avoids exposing forge tokens or other credentials to the gateway
container)
- tmpfiles: add /var/lib/hyperhive/gateway/ dir rule + seed agents.conf there
- scrub "(#869)" from hive-gateway-nginx-reload service description
Replace eval-time per-agent nginx location baking with a runtime
include file. c0re writes /var/lib/hyperhive/agents.conf (nginx
location blocks, UDS or TCP per agent) on every topology change and
on the 10s marker poll. The gateway container bind-mounts
/var/lib/hyperhive/ at /run/hive-state/ and nginx includes
/run/hive-state/agents.conf. A systemd path unit inside the container
watches the file for changes and fires `nginx -s reload` on each
atomic rename from c0re — no nixos-rebuild switch needed when agents
start, stop, or flip useUnixSocket.
- new hive-c0re/src/gateway_nginx.rs: write() + render()
- lib.rs + meta.rs + agent_sockets::spawn_poll: hook in write()
- hive-gateway.nix: drop agentPortsTable/agentSocketsTable/
agentUpstreamFor/lib.mapAttrs', add /run/hive-state bind-mount,
include directive, systemd path unit + reload service, tmpfiles
for /var/lib/hyperhive + agents.conf seed
- docs/gateway.md: update vhost table + Per-agent UDS section
Backend half of #726. The 32-line stderr ring buffer in
`lifecycle::run` and `lifecycle::prebuild_toplevel` routinely
truncated the actual eval error — a single 'tried alternatives'
block out of a failing module ref is often 30+ lines on its own,
which pushed the real cause out of the bailout message. With this
patch the full stream lands in sqlite where the dashboard can
surface it; bail-outs now point at the build log id instead of an
arbitrary tail.
### New module: `hive-c0re::build_logs`
`BuildLogs::open(db_path)` creates a sqlite db at
`<db_path>/build_logs.sqlite`. Schema: id, agent, kind, cmdline,
started_at, finished_at, status, stdout, stderr — indexed for
both per-agent latest-N queries and the status-driven retention
sweep. API: `start / append_stdout / append_stderr / finish` for
the streaming writer side (best-effort — every append handles
sqlite errors via tracing::warn so a transient blip never tears
down a rebuild), plus `list_recent_for_agent / get_full` for the
read side (50-row cap clamped server-side).
### Process-singleton handle
`build_logs::install / global()` install the `Arc<BuildLogs>` at
`Coordinator::open` so `lifecycle::run` and
`lifecycle::prebuild_toplevel` can write without us threading
the handle through every `pub async fn` entry point in the
lifecycle surface — there are 10+ call sites and the handle is
the same Arc everywhere anyway. Reads via `global()` return None
in early-startup / standalone-test paths so callers no-op cleanly.
### Lifecycle integration
`run` derives the kind from `args[0]` (the nixos-container verb)
and the agent name from `args[1]` (stripped of the `h-` agent
prefix so dashboard grouping matches the bare agent name). It opens
a row before spawning, pipes stdout/stderr into both tracing AND
the row, then `finish`es with the terminal status. `prebuild_toplevel`
does the same with kind = "prebuild" and the agent name already
in scope from its caller. On failure both bail with "see build log
#<id>" instead of the ring-buffer tail.
### Retention
`spawn_vacuum` mirrors `stats_vacuum`/`events_vacuum` in shape —
hourly tick that calls `BuildLogs::vacuum()`. Rule: failures kept
30d (operators dig into them), successes 24h (mostly noise after
a day), in-flight rows never reaped regardless of age (running
builds shouldn't disappear from their own log viewer mid-stream).
### Out of scope (follow-ups)
- Dashboard endpoints (`GET /api/build-logs/{agent}`,
`GET /api/build-logs/{id}`) — wire layer
- ContainerView.build_logs field — agent-card chip data source
- Side-panel viewer + SSE `build_log_appended` event — UX
- Download-as-text link — operator workflow polish
These all stack cleanly on top of the data layer + writer this
PR ships. Filing as phase 2 PRs.
### Validation
- 5 new unit tests pass (start/append/finish flow, list ordering
+ clamp, get_full miss, vacuum per-status rule, post-finish
append fault tolerance)
- 157 hive-c0re lib tests pass overall
- cargo check workspace clean
Refs #726.
When `services.hyperhive.gateway.enable` is on (default), the c0re
NixOS module now sets `HIVE_GATEWAY_ENABLED=1` on the service env.
`/api/state` exposes the flag as `gateway_enabled`; the SW4RM tab's
container-row renderer flips three link sites (primary agent-name
link, favicon fetch, `container`-kind nav-strip links) from the
legacy `http://<host>:<port>/` direct TCP shape to same-origin
`/agent/<name>/` URLs — the gateway proxies them to the per-agent
harness via `agent-ports.json` or `agent-sockets.json` (#784 / #815).
Gateway-off deploys keep the direct TCP fallback so local-dev /
operator opt-out keeps working.
`forge`-kind nav-strip links still resolve against `:3000` (separate
sub-domain transition, tracked by `forge.behindGateway`);
`external`-kind links are already absolute.
Mirrors the `HIVE_MATRIX_GUI_ENABLED` env→snapshot-flag pattern.
Docs updated: `docs/web-ui.md::Container row` + new
`docs/gateway.md::Dashboard link shape` section.
argus on PR #832:
- 🟡 spawn_poll was inserted BETWEEN write's closing doc and the
pub fn write line; rust treated the consecutive /// as one block,
so spawn_poll inherited write's tail and write ended up with no
closing doc. moved spawn_poll AFTER write to fix attribution.
- 🟡 spawn_poll(coord) took Arc<Coordinator> just to drop it
immediately. dropped the param; main.rs call site now just
agent_sockets::spawn_poll().
no functional change. 10 tests still pass.
closes the gate atlas raised on PR #813: without per-agent opt-in
signal, agent-sockets.json listed every sub-agent, and any agent
that hadn't flipped hyperhive.web.useUnixSocket would 502 the
gateway (its harness still binds TCP, no socket at the published
path).
harness side (web_ui::bind_unix):
- after successful bind + chmod, drop a `.bound` marker in the
per-agent dir as a stable 'this agent has a unix socket here'
signal. best-effort: a failed marker write logs at WARN but
doesn't abort serve (the socket still binds fine; gateway just
keeps using TCP for one more poll).
c0re side (agent_sockets):
- new READY_MARKER const + ready_marker_for(name) helper
- build_map filters by ready_marker_for(name).exists() — only agents
whose harness has bound the socket appear in the JSON map
- new build_map_with<F> internal extracts the predicate so tests
pass a controlled is_ready closure (no real fs access)
- new spawn_poll() background task: re-fires agent_sockets::write
every 10s so the JSON catches up to fresh markers without
needing a container-start hook. write() idempotency means
steady-state cost is one stat per agent per tick.
10 tests: 6 prior + new build_map_filters_by_ready_predicate +
ready_marker_path_is_sibling_of_socket. existing tests adjusted to
call build_map_with(_, |_| true) since the default path now hits
the fs.
once this lands + #822 lands, atlas's gateway-side step 3 can drop
its eval-time `pathExists` fallback — c0re only publishes opted-in
agents, so the gateway can trust the JSON unconditionally.
builds on step 2a (#809). lifecycle::set_nspawn_flags now adds a
--bind={socket_dir}:{socket_dir} flag per sub-agent so the harness's
HIVE_WEB_SOCKET bind (PR #800) lives in a dir both the agent
container and the host can see.
design (matches #809's a1a601d explanation):
- bind the SUBDIR, not the socket file. file bind-mounts drop on
unlink; the harness's bind_unix unlinks any stale socket before
binding, so a file bind would land the new socket in the agent's
private namespace, invisible to the gateway. dir bind keeps both
sides on the same dir inode.
- per-agent dir (one /run/hive-agent/<name>/ per agent, not a shared
/run/hive-agent/ mount). The agent's container only sees its own
subdir — never siblings' (mara on #800).
- manager skipped — the manager's UI serves at / via the c0re
dashboard upstream, not via /agent/<name>/, so it never needs the
per-agent socket dir.
mkdir source defensively before bind: nspawn refuses to start when
the bind source is missing, and /run/hive-agent/ doesn't exist on
fresh hosts.
remaining work in this phase:
- step 3 (atlas): gateway proxy_pass http://unix:/run/hive-agent/<name>/web.sock:/
- per-agent: flip HIVE_WEB_SOCKET in agent.nix to opt in (separate PRs)
- step 4 (later): drop TCP fallback once everyone's flipped
was: /run/hive-agent/<name>.sock (flat single-file bind-mount).
issue: file bind-mounts don't survive the harness's 'unlink stale
socket then bind(2) a new one' cycle. The unlink drops the bind
inside the container; the rebind happens in private container
namespace; host never sees the new inode → gateway can't connect.
now: /run/hive-agent/<name>/web.sock (per-agent SUBDIR + fixed
filename). Lifecycle bind-mounts the parent dir per agent (step 2b)
so both sides see the same dir inode; the socket appears on the
host the moment the harness binds it.
new helpers:
- AGENT_SOCKET_DIR const (parent, gateway binds this whole tree)
- SOCKET_FILENAME const ("web.sock")
- agent_dir_for(name) (per-agent subdir, lifecycle bind-mounts this)
- socket_path_for(name) (= agent_dir_for(name).join(SOCKET_FILENAME))
per-agent dir isolation also satisfies mara on #800 directly:
agent's container only sees its own subdir + socket, never siblings'.
8 tests now (added agent_dir_for_is_socket_parent invariant).
sibling to agent_ports.rs (#748 / #15). Writes
/var/lib/hyperhive/agent-sockets.json mapping agent name → unix
socket path under /run/hive-agent/<name>.sock — the source of truth
for the gateway's eventual unix-socket upstreams once it flips off
TCP per #784's transition plan.
shape mirrors agent_ports.rs:
- BTreeMap for deterministic JSON output
- atomic <path>.tmp + rename for partial-write safety
- idempotent (skip rewrite when content unchanged)
- MANAGER_NAME filter (gateway routes manager via dashboard upstream,
not /agent/<n>/)
socket_path_for(name) is pure-deterministic so a name alone resolves
to the same path the c0re writes + the harness binds via
HIVE_WEB_SOCKET (post-PR #800).
hooked into meta::sync_agents alongside the existing
agent_ports::write call (same best-effort + non-fatal shape).
next: step 2b (lifecycle bind-mount /run/hive-agent/ into agent +
gateway containers, with per-agent isolation per mara's #800 nudge
'agents can only access their own sockets'). then atlas's step 3
(gateway proxy_pass http://unix:...). then step 4 drops TCP.
