hive-ci is the one container on the hive bridge that used DHCP
(networking.interfaces.eth0.useDHCP = true, from #2310 / #2336). But the
bridge has no DHCP server: dnsmasq on the bridge is DNS-only and agent
containers get deterministic static IPs (lifecycle::agent_network_ip),
so the DHCP client never gets a lease and the container hangs at boot:
A start job is running for DHCP Client (5s / 1min 31s)
Assign a static address on eth0 (top host address of the subnet) plus a
default route via the bridge gateway, and drop useDHCP. Preserves the
#2310 netns isolation (no host-loopback reach) while letting the
container boot and reach the forge through the gateway.
Refs #2310, #2336.
The is_descendant_of and apply_set_parent cycle detection both used
hand-rolled 32-hop bounded ancestor walks. Correct in practice (no
real hive exceeds 32 levels) but carried an arbitrary ceiling and were
harder to reason about than proven graph primitives.
Changes:
- Add build_graph(): converts BTreeMap<name, parent|null> → DiGraph
with parent→child edges + BTreeMap<name, NodeIndex> index
- Add is_descendant_of_in(): pure (no disk I/O), uses
petgraph::algo::has_path_connecting from ancestor to candidate
- Rewrite is_descendant_of(): delegates to is_descendant_of_in(&read())
- Rewrite apply_set_parent() cycle detection: build_graph() + speculative
edge + is_cyclic_directed(); no depth limit
- Add tests for is_descendant_of_in (self, direct child, grandchild,
parent-is-not-child, sibling, unknown)
petgraph was already a workspace dep (used elsewhere). On-disk format
unchanged (flat JSON map). Public API surface unchanged.
nixos-containers.nix sets networking.useDHCP = false for every
container (to suppress the deprecated global DHCP warning). Setting
networking.useDHCP = true in the hive-ci inner config conflicts with
no priority differential → eval error on hosts running current main.
Fix: drop the global useDHCP = true and configure DHCP on eth0
(the inner veth interface name assigned by systemd-nspawn when
hostBridge is used). This is both conflict-free and the correct
modern NixOS practice for per-interface DHCP.
Claude's generic `system/status` subtype carries no detail beyond the
bare label, and previously each tick got its own note row. During a
compaction pass (which emits a burst of these with no other signal
before the completed compact_boundary) this looked like a wall of
identical noise followed by silence, making a routine compaction look
stuck.
Collapse consecutive status ticks into one updating row, same pattern
already used for the thinking_tokens counter.
The container-level nix.settings.fallback = lib.mkForce true (added as an
interim safety net so a down remote builder degrades gracefully rather than
hard-failing CI) is rendered redundant by the host-daemon routing fix: with
NIX_REMOTE=daemon set for all systemd units in the container, every nix
build invocation routes through the host daemon, which already has its own
fallback policy. The container setting would only apply to nix invocations
that bypass the daemon entirely — a case that should not exist post-deploy.
NOTE: draft — gated on the daemon-routing change (PR that adds
NIX_REMOTE=daemon to systemd.globalEnvironment + hive-priv service env)
being deployed and confirmed working in production before merging.
The knowledge repo is cloned with credentials embedded in .git/config;
those credentials survive on disk and the entire LOCAL_DIR (including
.git/) is bind-mounted read-only into every agent container. Any
agent (or prompt-injected PR build) can read the site-admin token via
cat /knowledge/.git/config.
Defense-in-depth layer: in write_nspawn_flags, append
--tmpfs=/knowledge/.git after the --bind-ro=.../knowledge flag whenever
the /knowledge bind mount is present. systemd-nspawn processes the tmpfs
after the bind, overlaying an empty in-memory directory over .git/ inside
the container. Agents see the working-tree documents but not the repo
metadata or stored credentials.
This is independent of the complementary fix in workers/knowledge.rs
(stop embedding credentials in .git/config in the first place). Both
layers together provide defense-in-depth: even if a future credential
accidentally reaches .git/config, it stays off the container floor.
Closes part of issue 2308 (bind-mount isolation layer).
Give hive-ci its own network namespace (privateNetwork=true, hostBridge)
instead of sharing the host netns. This closes the path by which a
prompt-injected PR nix build could reach the unauthenticated dashboard
at 127.0.0.1:7000.
The runner now reaches the forge through the gateway:
- networking.extraHosts maps forgeCfg.domain → bridgeIp
- runner url = http://${forgeCfg.domain} (gateway port 80, addSSL=true,
no HTTP→HTTPS redirect — HTTP is served alongside HTTPS)
- networking.useDHCP = true for the bridge-attached veth
- networking.nameservers = [bridgeIp] (dnsmasq on bridge, external fwd)
Ports 80 and 443 are already open on the bridge firewall (hive-network.nix).
The bridge→127.0.0.0/8 DROP rule (defence-in-depth) is unchanged — CI
can no longer reach 127.0.0.1:{7000 dashboard, forge httpPort, etc.}.
Requires forge.behindGateway=true (the default when hyperhive is enabled)
— asserted so misconfigured hosts get a clear build error.
Note: the host-side hive-ci-prefetch script still uses
127.0.0.1:${httpPort} directly (it runs on the host, not in the
container) — no change needed there.
reqwest has no UDS transport, so unix: upstreams dial the socket
directly with a raw hyper/1.1 client per request instead. http(s)://
upstreams are unaffected (still go through the existing reqwest path).
Adds hyper (client, http1), hyper-util (tokio IO adapter), and
http-body-util as direct hive-ag3nt dependencies - all three were
already present transitively via reqwest, this just uses them
directly for the new code path.
Agent containers currently mount /tmp as a 3.2 GB tmpfs (RAM-backed).
Large scratch writes (nix-develop shells, cargo build dirs) eat container
RAM unnecessarily and can exhaust the tmpfs, losing in-flight work.
Set boot.tmp.useTmpfs = false so /tmp is disk-backed instead.
boot.tmp.cleanOnBoot (default true) preserves the ephemeral-per-boot
semantics — /tmp is cleared on container start, same as before.
Closes#2264. Supersedes PR #2299 (TMPDIR redirect + state/tmp).
Both mdNode implementations (agent UI app.js, dashboard common.js)
assigned marked.parse() output straight to innerHTML with no
sanitizer. marked v5+ dropped its built-in sanitize option, and there
was no DOMPurify anywhere in frontend/, so markdown containing raw
HTML/script tags rendered live in the browser.
Both sinks receive untrusted input in practice: the agent UI's mdNode
renders recv tool_result bodies, assistant prose, and send/ask/answer
payloads sourced from peer agents and matrix-relayed messages (the
documented prompt-injection adversary); the dashboard's mdNode renders
agent-authored .md files served verbatim by GET /api/state-file
(the endpoint validates path, not content). Since the per-agent UI and
dashboard are same-origin behind the gateway with operator-authority
endpoints (approve/spawn/rebuild/destroy/answer-question), injected
script would run with the operator's session.
Fix: DOMPurify.sanitize() the marked.parse() output at both sinks
before assigning to innerHTML. Added dompurify as a dependency to
both the agent and dashboard npm workspaces, recomputed npmDepsHash
in nix/frontend.nix for the updated lockfile. Also corrected
docs/web-ui/shape.md, which claimed the markdown-rendering path was
XSS-safe by construction the same way the text-node-based linkify
path is — it isn't; it's safe because it's sanitized.
CSP hardening for the dashboard (no unsafe-inline) is a separate,
larger backend change (response headers in hive-c0re) and is left as
a fast-follow rather than folded into this fix.
Add [output] section to weston.ini with name=VNC-1 mode=1280x720.
The VNC backend's output is named VNC-1; the mode string sets the
compositor's framebuffer size that clients connect to.
On 409 (team already exists), list the org teams to find the operators
team id, then unconditionally PATCH to the desired settings via
org_edit_team. This self-heals a team that was created with the wrong
shape by an older code path (missing units, wrong permission) without
touching membership (separate endpoint, operator-managed).
Addresses mara's review: 'shouldnt we get, then change, then update'.
Unconditional PATCH is simpler than GET→diff→conditional PATCH and safe
here since we own units/permission/description fully.
mara: the background worker is redundant if c0re knows when its own
sockets go missing. damocles: 10s poll latency and redundancy are two
faces of the same issue — poll adds a reconnect window and does
redundant work when c0re could react directly.
design: c0re owns the MCP listener lifecycle, so the only time a
listener disappears without c0re knowing is when c0re itself restarts.
- replace spawn_poll (recurring 10s loop) with sync_on_start (one-shot
sweep at daemon boot): re-registers all running agents on startup
after /run/hyperhive/agents/ is cleared by the tmpfs reset.
- run_reconcile (reconcile-start path): add coord.register_agent(name)
immediately after start_with_fallback — event-driven, no poll delay.
- run_create already calls register_agent eagerly; kill/destroy paths
already call unregister_agent — no changes needed there.
tracker: #2290
Collapse the scattered ensure_agent_runtime_dir calls into the lifecycle
functions themselves so callers have a single responsibility:
- lifecycle::spawn: calls ensure_agent_runtime_dir before write_dropins.
Callers (handle_spawn, ensure_root_agent) no longer need a separate
preamble step.
- lifecycle::rebuild_no_meta spawn path: calls ensure_agent_runtime_dir
before write_dropins. apply_commit / merge_config_pr flows no longer
need a manual ensure_agent_runtime_dir.
- run_create (job-queue): drops ensure_agent_runtime_dir + register_agent.
The tail Reconcile's converge_start_preamble handles the runtime dir
and mcp_sockets::spawn_poll handles the listener. Create stays purely
'provision + create', not 'create + start'.
- handle_spawn (server.rs): drops manual preamble; lifecycle::spawn owns it.
Drops unneeded unregister_agent on failure (supervisor handles listener).
- ensure_root_agent (auto_update.rs): drops manual ensure_agent_runtime_dir.
- actions.rs apply_commit / merge_config_pr: drop manual
ensure_agent_runtime_dir; rebuild_no_meta's spawn path handles it.
Result: ensure_agent_runtime_dir lives in exactly two places —
lifecycle::spawn (direct spawn) and converge_start_preamble (start/reconcile
path). All other callers are clean call sites.
- lifecycle::StartableAgent: opaque token produced only by
converge_start_preamble. #[must_use] with a hint to call
start_with_fallback(token).
- lifecycle::converge_start_preamble(name, hive, paths): runs
ensure_agent_runtime_dir + write_dropins, returns StartableAgent.
The only way to obtain a token.
- lifecycle::start_with_fallback(token: StartableAgent): public API
now requires the token. Callers that skip the preamble get a compile
error, not a runtime outage.
- lifecycle::start_with_fallback_inner(name): private; used internally
by rebuild_no_meta where the preamble is already enforced structurally
(write_dropins was called on the line above).
- exec.rs ReconcileAction::Start: migrated to converge_start_preamble
+ start_with_fallback(token). The write_dropins + start_with_fallback
two-step is now a single typed pipeline.
- lifecycle::ensure_agent_runtime_dir(name): pure filesystem op, no
Coordinator dep. Creates /run/hyperhive/agents/<name> without touching
the MCP listener map.
- workers/mcp_sockets::spawn_poll(coord): 10 s reconcile loop (same shape
as agent_sockets::spawn_poll). Converges 'agent running => MCP listener
bound'. First tick is immediate so hive-c0re restarts re-register all
running agents without waiting a full interval. Fixes the dead-listener-
after-daemon-restart gap.
- All ensure_runtime() call sites updated:
- Prebuild/Swap/WriteDropin: Coordinator::agent_dir() (pure, no IO)
- Reconcile-Start: ensure_agent_runtime_dir + agent_dir (dir may be
missing after reboot; listener deferred to supervisor)
- run_create / handle_spawn: ensure_agent_runtime_dir + register_agent
(eager on first spawn so socket ready before harness first turn)
- apply_commit / merge_config_pr: ensure_agent_runtime_dir + agent_dir
- Manager (auto_update): ensure_agent_runtime_dir + agent_dir
(manager has no MCP listener; socket_server::start_manager owns it)
- ensure_runtime() retained in Coordinator with updated doc pointing at
the preferred split form. No callers remain outside tests.
Add a [Unit] section to hyperhive-limits.conf (the drop-in written by
write_resource_limits) with:
ConditionPathIsDirectory=/run/hyperhive/agents/<name>
When this condition is not met, systemd skips the unit with result
"condition" — NOT a failure, so the start-limit counter is not
incremented. Belt-and-braces on top of the tmpfiles.d fix (subtask 2):
if a dir is somehow absent at start time, the container idles instead of
restart-looping into start-limit-hit.
Also promote AGENT_RUNTIME_ROOT to a module-level const (was duplicated
inside two functions) and remove the duplicates.
systemd-tmpfiles d entries adjust mode+owner on existing dirs. Using
0755 root root would stomp live agents' socket dirs (owned by agent
uid:gid) on every sync_tmpfiles call, breaking the harness's ability
to bind new sockets until host_config rechowns them.
Fix: 0777 root root — matches the chmod_socket_dir(0o777) fallback
already used by host_config when the agent uid is unavailable. World-
writable dirs let the non-root harness bind sockets regardless of who
owns the dir. host_config's chown_socket_dir tightens ownership when
the agent uid is resolved.
Also add missing # Errors doc to priv_client::sync_agent_tmpfiles.
Root cause of the boot outage: container@h-* units try to start before
hive-c0re reaches ensure_runtime, so bind-mount source dirs are missing.
Fix: hive-c0re (via hive-priv, which runs as root) writes
/etc/tmpfiles.d/hyperhive-agents.conf whenever the agent set changes.
systemd-tmpfiles-setup.service (sysinit.target) reads it at every boot
BEFORE any container units start, pre-creating:
/run/hyperhive/agents/<name> — MCP socket dir (bind -> /run/hive)
/run/hive-agent/<name> — web socket dir (bind -> /run/hive-agent)
This alone removes the outage class: even if hive-c0re is slow to start,
the bind-mount sources exist and container units can activate.
Added:
- PrivRequest::SyncAgentTmpfiles { agents } in hive-sh4re
- sync_agent_tmpfiles() in hive-priv: generates content, writes atomically,
calls systemd-tmpfiles --create to apply immediately
- priv_client::sync_agent_tmpfiles() wrapper
- lifecycle::sync_tmpfiles() best-effort helper (list + priv call)
- Call sites: hive-c0re startup, handle_spawn success, destroy success
