When isolateContainers=true, claude (and all egress) broke in every
container: agents came up with an IP but no way off the bridge subnet.
Two container-side gaps, both confirmed against nixpkgs
nixos-containers.nix:
1. No default route. hive-priv wrote HOST_ADDRESS= empty in the nspawn
conf. nixos-container's container-side setup only installs
`ip route add default via $HOST_ADDRESS` when HOST_ADDRESS is
non-empty, so the container had an address but no gateway -> nothing
off-subnet (incl. api.anthropic.com) was reachable. Fix: write
HOST_ADDRESS=<bridge-ip>. In bridge mode the host-side address/route
setup is skipped, so this only affects the container's default route.
2. No usable resolver. nixos-container copies the host's /etc/resolv.conf
into the container at every start; the host resolver (e.g. 127.0.0.53)
is unreachable from a private netns and isn't authoritative for the
hive's own zones. Fix: hive-priv drops a marker carrying the gateway
IP only when isolated, and a new harness-base oneshot
(hyperhive-isolated-dns) rewrites resolv.conf to point at the bridge
dnsmasq. Inert in shared-netns mode (no marker), so the shared
container toplevel does the right thing in both modes.
The gateway IP is the address part of HIVE_NETWORK_SUBNET (the bridge IP
verbatim, honouring a non-.1 operator override), via a new validated
bridge_gateway_ip() helper with unit tests.
Unblocks defaulting isolation on.
Clean the legacy backlog so the tracker-tag lint can become a required
gate (mara's warn-during-cleanup -> full-tree-enforcement path). Rewrite
the ~33 real `closes/see #NNN` provenance refs in doc-comments to prose
across hive-forge, hive-c0re, hive-ag3nt, hive-matrix-mcp, hive-sh4re,
and add a `lint:allow` line marker to check-issue-refs.sh for genuine
non-tracker `#<digits>` (a hash-digit heading-detection test input).
Tree is now lint-clean; tracker-tag lint ready to promote to required.
The gateway container starts alongside every hyperhive deployment, so
gating it behind a separate enable flag was a footgun: an operator who
set it false lost the only thing exposed to the outside while the
agent containers kept running. Re-gate the gateway config on the
top-level services.hyperhive.enable instead.
- hive-gateway.nix: drop the gateway.enable mkOption; gate the config
block on config.services.hyperhive.enable.
- hive-forge.nix: behindGateway now defaults to services.hyperhive.enable;
remove the behindGateway-requires-gateway assertion (now vacuous).
- hive-network.nix: remove both gateway.enable assertions (vacuous).
- hive-c0re.nix: drop the firewall.allowedTCPPortRanges 8100-8999
fallback that opened agent ports when the gateway was off (the
gateway is now the sole entry point); HIVE_GATEWAY_ENABLED is always
set since the gateway always runs.
- nix/docs/default.nix: remove the gateway.enable = mkForce false stub
(would be an eval error against the removed option; the gateway is
already re-gated on hyperhive.enable, which docs force false).
- hive-matrix.nix, dashboard.rs: comment/prose updates only.
BREAKING: operators relying on services.hyperhive.gateway.enable = false
to suppress the gateway must instead point their own reverse proxy at
the gateway's port. NixOS errors clearly on the now-unknown option.
The systemd ExecStart carried every host-level setting as its own flag —
nine of them, including two escaped JSON blobs (the context-window map and
the model-price table). Collapse them into a single `--config <file>` JSON.
- Reuse the existing HiveEnv as the container-injected config shape (add
Deserialize + Default), and add a ServeConfig wrapper = flattened HiveEnv
plus the hive-c0re-local model_prices table (kept out of HiveEnv since it
is never injected into containers). serde(default) lets any field be
omitted and fall back to its canonical default.
- clap: add --config; the per-setting flags become optional overrides
(config file is the base, explicit flags win — preserves hivectl/debug
ergonomics and bare `hive-c0re serve`).
- Coordinator::open and cmd_serve now take the bundled HiveEnv, which drops
their too_many_arguments clippy allows. cmd_serve keeps a single
too_many_lines allow (inherent daemon-boot orchestration, not arg-driven).
- nix: write the config as JSON to the store + pass --config, so ExecStart
is one short line.
- Add a round-trip test proving the flatten + per-field defaults work.
Closes the ExecStart-length issue.
Per operator request: instead of a disk-specific alert, surface a generic
server-warnings banner at the very top of every page, so new system
warnings can be added backend-side with no frontend change.
- hive-c0re `host_stats`: `server_warnings() -> Vec<ServerWarning>`
(`{ kind, level, message }`). The threshold logic lives server-side; the
host disk-pressure check (a `statvfs` probe of `/nix`: ≥85% used → warn,
≥95% → crit) is the first and only producer today. No new deps (libc).
- `/api/state` carries `server_warnings` (replaces the disk-specific
field). Empty when all clear.
- frontend: `renderServerWarnings` / `initServerWarnings` in `common.js`
inject a sticky top-of-<body> banner and render the list, coloured by
`level`. Wired on every page — dashboard (live, via refreshState),
FL0W, L0GS, H0M3. No per-warning frontend code; adding a warning kind
is a pure backend change.
cargo check/clippy/fmt + npm run build green. Closes#1518.
Swarm-level companion to the per-agent favorite-tools doughnut (#1433).
Aggregates each agent's bash_commands(ts, head) table (written by
hive-bash-mcp) across the whole hive and surfaces the top-10 most-run
command heads on the dashboard ST4TS tab, alongside the existing model
mix.
- hive_stats.rs: AgentAgg gains a `bash` head→count map, filled by a new
guarded `read_bash_heads()` that reuses read_agent's read-only
connection. A missing `bash_commands` table (capture hasn't run for
that agent) or any read error yields an empty map — isolated from
read_agent's error path so it never drops an agent from the rollup.
HiveStats gains `bash_mix: Vec<KeyCount>` (busiest-first, top 10).
Unit tests cover the per-head tally + window cutoff and the
missing-table degrade-to-empty path (in-memory sqlite).
- dashboard: a "favorite tools (bash commands across the swarm)" CSS-bar
list on the ST4TS pane, mirroring the model-mix bars. Header + list
stay hidden until bash_mix has data, so a fresh hive shows no empty
block. (Dashboard ships no chart lib — bars, not a doughnut.)
- docs: dashboard.md ST4TS section documents the new rollup.
Closes#1449. Inert until the hive-bash-mcp capture (#1448, merged) has
recorded data across agents.
Per operator request on the PR: the built-in/default prices were the old
Claude 3 numbers (opus 15/75, etc.). Update opus + haiku to the current
Claude 4.x family list pricing (cache_write = the default 5-minute cache
TTL); sonnet was already correct:
- opus: input 5, output 25, cache_read 0.5, cache_write 6.25
- sonnet: input 3, output 15, cache_read 0.3, cache_write 3.75 (unchanged)
- haiku: input 1, output 5, cache_read 0.1, cache_write 1.25
Updated in both builtin_prices (hive_stats.rs) and the nix modelPrices
default (hive-c0re.nix), with cross-reference "keep in sync" notes on
both sides. Also addresses the earlier reviewer note: dropped the
over-strong "single source of truth" wording in the --model-prices arg
doc (the nix default does mirror the numbers in production).
- modelPrices submodule fields use lib.types.numbers.nonnegative
instead of lib.types.float: accepts bare ints (15) as well as floats
(15.0) and rejects negative prices for free.
- Collapse the triple-sourced default: hive-c0re serve --model-prices
now defaults to "{}" so builtin_prices() is the single in-code
fallback. The nix option default still carries the full
opus/sonnet/haiku table to self-document prices for operators.
The hive-wide cost estimate on the dashboard's ST4TS tab used a
hard-coded model->price table in hive_stats.rs. Anthropic list pricing
drifts, so move the table to a nix option operators can keep current
without a code change.
- New `services.hyperhive.modelPrices` option: attrset of model-family
short name -> { input, output, cache_read, cache_write } USD per
million tokens. Passed to `hive-c0re serve --model-prices <json>`.
- hive_stats: `Prices` is now public + Deserialize; add `PriceTable`
type and `resolve_prices` (longest case-insensitive substring key
wins) with the old hard-coded table preserved as `builtin_prices`
fallback for any model not covered.
- Coordinator holds the parsed table (hive-c0re-local, not injected
into containers, so not part of HiveEnv); `/api/stats-hive` reads it.
- Docs: dashboard.md ST4TS cost note updated; option self-documents
via nixosOptionsDoc.
Closes#1434
The operator-inbox handler landed with an unformatted long match line
(`match state.coord.broker.unread_for_recipient(...)`) that rustfmt
wants wrapped. It was force-merged during the CI outage so the formatting
gate didn't catch it, and now the treefmt check fails for every PR based
on current main. Pure formatting, no logic change.
Agents that `send(to: "operator")` were easy to miss — they only
surfaced on the FL0W firehose with no read-state (#1469). Surface them
on the Y3R C4LL ("things waiting on you") tab as a proper inbox.
Backend:
- broker: `unread_for_recipient(recipient, limit)` — unacked messages
for a recipient, newest-first. Mirrors `mark_all_read`'s filter
EXACTLY (`recipient = ?1 AND acked_at IS NULL`, no `delivered_at`
condition) so everything listed is exactly what mark-read clears —
operator rows never get `delivered_at` set (no agent-socket recv).
- dashboard: `GET /api/operator-inbox` → `{ messages: [...] }` (id,
from, body, at, in_reply_to, validated file_refs). Mark-read reuses
the existing `POST /api/agent/operator/mark-all-read` (the route
format-validates the name; "operator" passes; `mark_all_read`
already acks `to="operator"` rows).
Frontend (Y3R C4LL):
- New ◆ 1NB0X ◆ section listing unread messages (sender · time · body,
path-linkified) + a "✓ mark all read" button.
- Cold-loaded on page load + on tab activation; appended live from the
broker `sent` stream (deduped on row id); cleared on mark-all-read.
- Unread count folds into the Y3R C4LL tab pill + the browser-title
`(N)` prefix, so messages are visible from any tab.
Removing the now-redundant FL0W operator-inbox UI is a clean follow-up
(deferred to avoid a flow.js conflict with the in-flight #1473).
Backend (broker + route) is host-side — @damocles to review per plan.
Closes#1469.
The restricted join rule needs room version >= 8. Pin it explicitly so a
future homeserver-default change can't silently invalidate the restricted
rule (which would fall back to invite-only and break the operator-joins-
from-the-Space path). Per review on the PR.
The hive Space was created empty — joining it surfaced no rooms because
Matrix doesn't auto-join a Space's children. Provision a default
"hive-chat" room on the matrix sweep, wire it bidirectionally to the
Space (m.space.child on the Space, m.space.parent on the room), and
invite @hive + every agent. The room uses a restricted join rule
allowing any Space member to join, so the operator (a Space member) can
join it from the Space hierarchy without an explicit invite.
Idempotent, mirroring ensure_hive_space: persisted chat-room-id wins,
else rediscover a non-space room named hive-chat, else createRoom. The
space-child link is re-applied each sweep (idempotent PUT) so a
recovered room reconverges its hierarchy link. Room id persisted to
matrix/chat-room-id (0600, survives destroy --purge).
Per review: available_parallelism() respects the hive-core process's
CPU affinity, so if it's ever affinity-pinned the denominator would
under-count and inflate cpu_pct. Read the online host CPUs from
/sys/devices/system/cpu/online instead (fall back to the process count,
then 1) so the 'percent of total host CPU' definition holds regardless.
C0NT41N3R L04D on the SYST3M tab + GET /api/container-resources.
Backend (hive-c0re/src/container_stats.rs): reads cgroup v2 cpu.stat +
memory.{current,peak,max} for each running agent machine
(machine-h\x2d<name>.scope under machine.slice), read-only/world-
readable so no hive-priv. CPU is a two-sample (~200ms) host-normalised
percentage; one shared sleep covers all agents. Skips agents whose
scope dir is absent (= not running). Network omitted: agents share the
host netns, no per-container counter.
Frontend: a polled C0NT41N3R L04D table on SYST3M (agent / cpu / mem /
peak / limit with meter bars), reusing the ST4TS table style. Polls
/api/container-resources every 5s only while the tab is active.
Backend reviewed-in-principle by damocles (path escaping + cpu delta
math); ping for the on-host sign-off.
Builds the dashboard UI on the /api/stats-hive endpoint: a new ST4TS
tab showing swarm totals (active agents, turns, tokens, labelled est
cost), a busiest-first per-agent table, and a model-mix bar list.
Plain tables/CSS bars — no chart lib in the dashboard bundle; per-agent
trend charts stay on each agent's own /stats page. Fetched on tab
activation + window change (pull-only, no SSE).
Also adds conn.busy_timeout(500ms) in hive_stats read_agent per review:
turn_stats is rollback-journal, so a read landing mid-INSERT would hit
SQLITE_BUSY and silently drop that active agent — wait the write out.
#1424 P2 backend. New hive_stats module aggregates every agent's
turn-stats.sqlite read-only (reusing Coordinator::kept_state_names +
agent_harness_dir, skipping missing/unreadable dbs) into swarm totals,
a busiest-first per-agent rollup, swarm model mix, and a labelled USD
cost estimate (rough model->price table; can move to a nix option
later). Exposed as GET /api/stats-hive?window=. Dashboard UI follows.
