Follow-up to the harness fix: any unit whose ExecStart is a generated
wrapper (a NixOS `script =` block or a writeShellScript) gets its journal
SyslogIdentifier derived from the wrapper's store-path basename — an
opaque, unstable name. Pin an explicit stable identifier on every unit
hyperhive declares.
script/wrapper units (leaked store paths): forgejo-gpg-init,
hive-gateway-self-signed-cert, hive-tls-ca, hive-ci-prefetch,
hyperhive-isolated-dns, tea-login, forge-avatar-sync,
matrix-avatar-sync, hive-dashboard-links.
direct-binary units (basename already sane; explicit for tidiness +
future-proofing): hive-c0re, hive-priv, hive-matrix-daemon,
hive-bash-daemon, weston.
Only changes the displayed line identifier; journalctl -u <unit>
filtering (keys on _SYSTEMD_UNIT) is unaffected.
`nixos-container update <agent>` (run by the hive-priv root helper) invokes
nix, which shells out by bare name to `git` whenever it has to fetch or
re-resolve a git-source flake input, and to `ssh` to dispatch builds to
remote builders. The hive-priv service `path` had neither, so an agent whose
config carries a git-source input — or a stale flake.lock whose node URL no
longer matches the flake's declared input, forcing nix to re-resolve at eval
— fails the build with `executing "git": No such file or directory`, and
remote-builder dispatch fails with `Could not find executable 'ssh'`.
Add `pkgs.gitMinimal` + `pkgs.openssh` to the path. Latent-bug fix: nix
legitimately needs both for any git-source input / remote build. Surfaced by
the bitburner agent's build failing on a git-source input re-resolution.
The internal forge is the canonical store for the meta flake, every
agent's config repo, and the internal/* repos, so it can no longer be
optional. Remove the services.hyperhive.forge.enable toggle:
- hive-forge.nix: drop the `enable` option; the forge config now
deploys gated on `services.hyperhive.enable` (it ships with hyperhive).
- hive-c0re.nix: HIVE_FORGE_URL env unconditional; forge-public-URL gate
drops the enable check (keeps behindGateway).
- hive-gateway.nix: local /etc/hosts forge entry keyed on behindGateway.
- hive-ci.nix: drop the now-moot `forge.ci.enable requires forge.enable`
assertion (forge is always present); reword the option doc.
- nix/docs/default.nix: drop the `forge.enable = mkForce false` stub
(option gone); the options-doc eval stays light via hyperhive.enable.
- hive-c0re forge.rs / hivectl.rs: reword 'forge.enable = true' error
text to 'wait for hive-c0re to start the container' (the runtime
token-absent path is unchanged — it's a bootstrap-timing check, not
the opt-out).
- docs/approvals.md, docs/ci.md: drop stale forge.enable references.
Migration: configs that set `services.hyperhive.forge.enable = false`
must drop the line — the forge is now mandatory.
Prereq/companion to #1838 (PR-based config flow, which assumes the forge
is always present).
Add swarm.peers.<domain>.caCert (path to a peer hive's root CA PEM),
trusted everywhere the hive's own internal CA is — so a self-signed
peer hive can federate (matrix) and any in-hive consumer validates its
certs.
Mechanism (reuses the existing hive-CA embedding): the meta-flake
renderer embeds a LIST of CA files next to each agent's flake —
hive-ca.pem (the hive's own self-signed CA, when active) plus each peer
caCert as peer-ca-<N>.pem — and emits them all in
security.pki.certificateFiles, so every agent trusts them at build
time. The matrix container trusts the same peer CAs for federation TLS.
Nothing is installed in the host trust store; the certs live in the nix
store (no mutable host file).
- meta.rs: embedded_ca_files() = hive CA + peer CAs (from new
HIVE_PEER_CA_PATHS env); ca_embed_state() tracks the list (content +
add/remove); sync_agents materialises + stages the list; render emits
the multi-entry certificateFiles. Tests cover hive-only / hive+peers
/ peers-only / none.
- hive-c0re.nix: HIVE_PEER_CA_PATHS service env (colon-joined caCerts);
caCert / certFingerprint option docs updated to the hive-wide scope.
- hive-matrix.nix + docs/swarm.md: scope + comment updates.
certFingerprint stays the c0re-only leaf-pin path.
nginx proxied `<hive>/` straight to hive-c0re:7000, and hive-c0re served the
dashboard dist itself via `tower_http::ServeDir` (from `HIVE_STATIC_DIR` baked
into its service env). So a frontend-only change rebuilt the hive-c0re unit and
restarted the core daemon — every operator session dropped its SSE stream for a
pure CSS/JS change.
The gateway nginx now static-serves the dashboard dist directly; hive-c0re's
dashboard router is API-only. The split uses the Accept-header SPA fallback (the
same `map $http_accept` pattern the matrix/agent vhosts already use), so no
backend prefix has to be enumerated: a browser navigation (Accept: text/html)
whose path is not an on-disk asset gets the SPA index.html; everything else
(every /api route, the bare action/mutation routes, the two SSE streams, the
knowledge webhook — all Accept != text/html) falls through `try_files` to the
`@c0re` named location and is reverse-proxied to hive-c0re. A new c0re route
needs no gateway change.
- hive-c0re.nix: expose the themed dist as a new internal read-only option
`services.hyperhive.c0re.servedFrontend`; drop `HIVE_STATIC_DIR` from the
service env (the router no longer serves files).
- hive-gateway.nix: read that option in host-module scope (dashboardDist),
static-serve `dashboard/` with the Accept-header `try_files ... @c0re` split;
`@c0re` carries `proxy_buffering off` + a 1d read timeout for the SSE streams
and a duplicated auth_basic block (named locations do not inherit it). The
dashboard map is unconditional; the matrix map stays gated on the matrix GUI.
- dashboard.rs: drop the ServeDir fallback + the HIVE_STATIC_DIR resolution; the
router 404s unmatched paths (the gateway only proxies non-static requests).
- hive-c0re/Cargo.toml: drop the now-unused tower-http dependency.
- docs/gateway.md: document the dashboard static split + the `@c0re` fall-through.
The store path is reachable inside the gateway nspawn container (shared
/nix/store), mirroring how HIVE_AGENT_FRONTEND_DIR already exposes the per-agent
UIs. The gateway and c0re changes must land together (atomic cutover) or the
dashboard 404s — this needs a watched gateway + c0re rebuild.
Progressive enhancement: a brand-new agent's state root under
/var/lib/hyperhive/agents is created as a btrfs subvolume when the host
filesystem is btrfs, otherwise it falls back to a plain directory. No
existing agent is auto-migrated — the new path only fires when the root
does not yet exist, so plain-dir agents are left untouched until an
explicit opt-in upgrade.
Two new privileged ops (subvolume create/delete are root-only):
EnsureAgentSubvolume statfs-gates on btrfs, creates the subvolume, and
chowns it to the hive-core user so the normal state/claude/harness
mkdirs succeed inside it; DeleteAgentSubvolume btrfs-subvolume-deletes
the root iff it is actually a subvolume. hive-c0re calls Ensure before
the per-agent dirs are created (spawn/rebuild/InitConfig) and Delete on
the purge path only — destroy keeps the subvolume for revival, matching
plain-dir semantics. btrfs-progs added to the hive-priv unit PATH.
Per-subvolume usage accounting + optional quota is a separate
follow-up.
Replace the gateway's bare in-container self-signed leaf with a leaf
signed by a host-held hive CA. A bare self-signed leaf is its own trust
anchor, so every regeneration is a fresh anchor every consumer would have
to re-trust, and a runtime-generated in-container leaf cannot be wired
into an agent's build-time trust store at all. A stable CA fixes both: a
single anchor that agents and federation peers trust once, surviving leaf
rotation.
New hive-tls module: a host oneshot generates a long-lived CA (default
~20y) under services.hyperhive.tls.stateDir and signs a gateway leaf
(default ~10y, SAN covering the bare domain, forge., matrix. and the
wildcard). It is ordered before the gateway container so the leaf exists
when nginx starts. Active only when the gateway uses self-signed TLS
(default) and a domain is set; inert under operator-cert or ACME modes.
Gateway: bind-mount the host CA dir read-only at /run/hive-ca; the
existing in-container cert unit now imports the host leaf into nginx's
state dir (copy as root, key left root:nginx 0640 for the pre-start
config test) instead of generating one. Cert/key paths nginx serves are
unchanged.
Foundational step toward agent + federation trust of self-signed hives;
no behaviour change for agents yet (they still reach the forge over plain
http on port 80). Eval-proven across self-signed, certDir and the inert
default paths.
Per mara's review of #1718: the HIVE_FORGE_URL / HIVE_MATRIX_URL comments
carried "network isolation is now unconditional / shared-netns removed"
backstory that doesn't belong on those keys. Trimmed to describe just the
value (the gateway vhost) + the gatewayHost null-guard. The
isolation-removal narrative stays where it's on-topic — the network
module's own deprecated-option descriptions and warning text.
Per mara's review of #1718: this env block only exists when hyperhive is
enabled (it lives inside `config = mkIf cfg.enable`), so the
`if hyperhive.enable then gateway else loopback` branch was dead — the
loopback fallback could never be reached. Set HIVE_FORGE_URL to the
gateway vhost directly.
HIVE_MATRIX_URL drops the same redundant `hyperhive.enable` term but
keeps the gatewayHost null-guard — that one is a real fallback for a
domain-less config, not the dead enable branch.
Addresses argus's review of #1718: the HIVE_FORGE_URL / HIVE_MATRIX_URL
conditionals still keyed off the now-deprecated `network.enable`, so
setting it to `false` would point agents at host loopback (unreachable
from their private netns) even though isolation stays on — contradicting
the "deprecated and ignored" warning. Guard on the top-level
`hyperhive.enable` instead, so `network.enable` truly has no effect
anywhere and the loopback branch only covers a hyperhive-disabled host.
Eval-proven: with `network.enable = false`, HIVE_FORGE_URL now resolves
to the gateway vhost (`http://forge.<domain>`), and the deprecation
warning still fires.
Network isolation is validated working end to end (forge + matrix reach
the gateway under private netns), so the shared-host-netns mode is
removed — there is now one mode.
- hive-network.nix: the bridge/DNS block and the isolation overlay are
now unconditional whenever hyperhive is enabled (gated on
services.hyperhive.enable, not the per-feature toggles). The redundant
isolateContainers-implies-enable guard is dropped.
- network.enable and isolateContainers are kept as DEPRECATED no-op
options so existing configs that set them to true still evaluate
cleanly; setting either to false now warns (gated on hyperhive being
enabled, so a non-hyperhive host gets no spurious warning) and has no
effect. Both are slated for removal in a future release.
- hive-c0re.nix: the HIVE_FORGE_URL / HIVE_MATRIX_URL conditionals drop
the "&& isolateContainers" clause — agents always reach forge/matrix
through the gateway vhost now.
- hive-matrix.nix: refresh one stale comment.
The Rust lifecycle is env-driven (HIVE_NETWORK_ISOLATION), so no Rust
change is needed: the env is now always set, so the existing
private-netns path always runs.
Supersedes the default-on flip (the option it would have defaulted is
gone).
Part of the tracker-tag cleanup: the hive convention is prose, not
issue-tracker tags, in code. Reword the 21 tags in the nix tree
(flake.nix + the hive-c0re/ci/gateway/network modules) to describe
the thing they pointed at, preserving the context without the tag.
Comment-only — no eval or logic change. Validated with nix fmt
(no reformatting) and nix flake check --no-build (all checks
evaluate clean); the full build check was skipped locally because
the shared remote builder is degraded, so CI will exercise the
build derivations once the runner recovers.
The gateway container starts alongside every hyperhive deployment, so
gating it behind a separate enable flag was a footgun: an operator who
set it false lost the only thing exposed to the outside while the
agent containers kept running. Re-gate the gateway config on the
top-level services.hyperhive.enable instead.
- hive-gateway.nix: drop the gateway.enable mkOption; gate the config
block on config.services.hyperhive.enable.
- hive-forge.nix: behindGateway now defaults to services.hyperhive.enable;
remove the behindGateway-requires-gateway assertion (now vacuous).
- hive-network.nix: remove both gateway.enable assertions (vacuous).
- hive-c0re.nix: drop the firewall.allowedTCPPortRanges 8100-8999
fallback that opened agent ports when the gateway was off (the
gateway is now the sole entry point); HIVE_GATEWAY_ENABLED is always
set since the gateway always runs.
- nix/docs/default.nix: remove the gateway.enable = mkForce false stub
(would be an eval error against the removed option; the gateway is
already re-gated on hyperhive.enable, which docs force false).
- hive-matrix.nix, dashboard.rs: comment/prose updates only.
BREAKING: operators relying on services.hyperhive.gateway.enable = false
to suppress the gateway must instead point their own reverse proxy at
the gateway's port. NixOS errors clearly on the now-unknown option.
Builds on the merged dashboard theming: the themed-frontend overlay now
swaps colors.css in both the dashboard/ and agent/ dist subtrees, and the
agent frontend dir (HIVE_AGENT_FRONTEND_DIR, which the gateway serves agent
static files from) points at the themed tree too. So when stylix is enabled
on the host, both the dashboard and the gateway-served agent UIs re-theme
from the host palette — still zero-op auto-detect, still a no-op (both dirs
on the unthemed dist) when stylix isn't imported.
Not covered: an agent reached directly on its own harness web server (no
gateway) serves from its per-agent mergedDist, built in the agent's own
nixosSystem with no host stylix access — that path needs the base16 palette
forwarded host->agent and is tracked separately.
When the operator's host config has stylix enabled, generate a base16
colors.css from its palette and overlay it onto the bundled frontend dist,
then serve that themed tree as the dashboard's HIVE_STATIC_DIR. Zero-op
auto-detect — no flake input, no operator action, no npm/esbuild rebuild
(a pure file-copy over the prebuilt dist). colors.css is the whole swap
contract; theme.css derives the semantic vars from the 16 base16 slots.
Guarded access (config.lib.stylix / config.stylix.enable via 'or' fallbacks)
makes it a clean no-op when stylix isn't imported — HIVE_STATIC_DIR stays
on the unthemed dist, which is what every non-stylix deployment + CI gets.
Dashboard surface only for now; the agent surface needs the base16 palette
forwarded host->agent (the per-agent harness builds its own dist) and is a
separate follow-up.
The systemd ExecStart carried every host-level setting as its own flag —
nine of them, including two escaped JSON blobs (the context-window map and
the model-price table). Collapse them into a single `--config <file>` JSON.
- Reuse the existing HiveEnv as the container-injected config shape (add
Deserialize + Default), and add a ServeConfig wrapper = flattened HiveEnv
plus the hive-c0re-local model_prices table (kept out of HiveEnv since it
is never injected into containers). serde(default) lets any field be
omitted and fall back to its canonical default.
- clap: add --config; the per-setting flags become optional overrides
(config file is the base, explicit flags win — preserves hivectl/debug
ergonomics and bare `hive-c0re serve`).
- Coordinator::open and cmd_serve now take the bundled HiveEnv, which drops
their too_many_arguments clippy allows. cmd_serve keeps a single
too_many_lines allow (inherent daemon-boot orchestration, not arg-driven).
- nix: write the config as JSON to the store + pass --config, so ExecStart
is one short line.
- Add a round-trip test proving the flatten + per-field defaults work.
Closes the ExecStart-length issue.
Two small follow-ups after the modelPrices PR (#1442):
- nix/modules/hive-c0re.nix: the `modelPrices` option `example` still
showed the old 5-minute sonnet `cache_write = 3.75`; the default is now
the 1-hour TTL `6.0`. Bump the example to match so it doesn't mislead.
- frontend/packages/agent/agent.css: remove the dead `.stats-empty-note`
selector (no element uses it — the stats empty state paints on canvas).
Per operator request on the PR: the built-in/default prices were the old
Claude 3 numbers (opus 15/75, etc.). Update opus + haiku to the current
Claude 4.x family list pricing (cache_write = the default 5-minute cache
TTL); sonnet was already correct:
- opus: input 5, output 25, cache_read 0.5, cache_write 6.25
- sonnet: input 3, output 15, cache_read 0.3, cache_write 3.75 (unchanged)
- haiku: input 1, output 5, cache_read 0.1, cache_write 1.25
Updated in both builtin_prices (hive_stats.rs) and the nix modelPrices
default (hive-c0re.nix), with cross-reference "keep in sync" notes on
both sides. Also addresses the earlier reviewer note: dropped the
over-strong "single source of truth" wording in the --model-prices arg
doc (the nix default does mirror the numbers in production).
- modelPrices submodule fields use lib.types.numbers.nonnegative
instead of lib.types.float: accepts bare ints (15) as well as floats
(15.0) and rejects negative prices for free.
- Collapse the triple-sourced default: hive-c0re serve --model-prices
now defaults to "{}" so builtin_prices() is the single in-code
fallback. The nix option default still carries the full
opus/sonnet/haiku table to self-document prices for operators.
The hive-wide cost estimate on the dashboard's ST4TS tab used a
hard-coded model->price table in hive_stats.rs. Anthropic list pricing
drifts, so move the table to a nix option operators can keep current
without a code change.
- New `services.hyperhive.modelPrices` option: attrset of model-family
short name -> { input, output, cache_read, cache_write } USD per
million tokens. Passed to `hive-c0re serve --model-prices <json>`.
- hive_stats: `Prices` is now public + Deserialize; add `PriceTable`
type and `resolve_prices` (longest case-insensitive substring key
wins) with the old hard-coded table preserved as `builtin_prices`
fallback for any model not covered.
- Coordinator holds the parsed table (hive-c0re-local, not injected
into containers, so not part of HiveEnv); `/api/stats-hive` reads it.
- Docs: dashboard.md ST4TS cost note updated; option self-documents
via nixosOptionsDoc.
Closes#1434
WriteAgentForgeToken and WriteAgentMatrixToken write agent state files
under /var/lib/hyperhive/agents/<name>/state/. The hive-priv service
has ProtectSystem=strict which makes all of /var read-only by default,
but /var/lib/hyperhive was missing from ReadWritePaths, causing EROFS
on every token write.
Closes#1274.
With ProtectSystem=strict (added in the previous commit), / is read-only
inside hive-priv. nix creates a temp result symlink in its cwd; without
WorkingDirectory that cwd is / (systemd default), causing:
error: creating symlink /.tmp.tmp-...: Read-only file system
Point WorkingDirectory at the StateDirectory (/var/lib/hive-priv) which is
always writable under strict. Nix drops its temp symlink there instead.
when isolateContainers=true, isolated agents have dnsmasq as their
resolver — forge.<domain> resolves to bridgeIp. route HIVE_FORGE_URL
through nginx on port 80 instead of exposing the raw forge port.
- HIVE_FORGE_URL: http://<forge.domain> when isolated (nginx proxies)
- bridge firewall: open 80+443 for agents to reach nginx (gateway)
- remove forge-specific httpPort rule (no longer needed)
- update docs/gateway.md + docs/network.md
per mara's review comment on PR #1150.
When containers run in private netns (isolateContainers=true), host
loopback is unreachable so HIVE_FORGE_URL=http://127.0.0.1:3000 breaks.
- nix/modules/hive-network.nix: when isolateContainers is on + forge
is enabled, open forge.httpPort on the bridge interface so agents
can reach forgejo at bridgeIp:httpPort (forgejo binds 0.0.0.0)
- nix/modules/hive-c0re.nix: HIVE_FORGE_URL switches to bridge IP
when network.enable && isolateContainers; loopback path retained
when isolateContainers=false
- docs/network.md: add Forge access + Forge URL rows to effects table
- docs/gateway.md: rewrite HIVE_FORGE_URL section for both modes
Add opt-in WireGuard mesh support to services.hyperhive.swarm:
- swarm.peers.<domain>.wireguardPublicKey — peer's wg public key
- swarm.peers.<domain>.wireguardEndpoint — peer's UDP endpoint (optional)
- swarm.peers.<domain>.wireguardAddress — peer's mesh IP with prefix
- swarm.wireguard.enable — bring up wg-hive interface
- swarm.wireguard.privateKeyFile — path to host's wg private key
- swarm.wireguard.address — this host's mesh IP/prefix
- swarm.wireguard.listenPort — UDP listen port (default 51820)
- swarm.wireguard.persistentKeepalive — keepalive seconds (default 25)
When enabled, generates networking.wireguard.interfaces.wg-hive with
one peer entry per mesh-enabled swarm.peers entry. Opens listenPort
UDP on the host firewall. Adds wireguard_address to HYPERHIVE_PEERS
JSON so hive-c0re can use mesh IPs for intra-swarm routing.
Assertions guard against enable=true without privateKeyFile or address.
Also refactors networking.firewall.allowedTCPPortRanges from the
nested attrset form (which conflicted with the new allowedUDPPorts
line) to the per-attribute form.
docs/swarm.md: adds WireGuard setup section with key generation
commands, two-hive config example, NAT/keepalive notes.
Privsep splits repo ownership: hive-priv (root) fetches the hive-core-owned
meta/applied repos via nix, and hive-c0re (hive-core) fetches the agent-owned
proposed-config repos. git/libgit2's dubious-ownership guard rejects both
('repository path is not owned by current user'), failing every rebuild.
Install a root-trusted gitconfig with safe.directory=* in each service's
HOME; both already have read access — this only satisfies the guard.
Two privsep follow-ups, both in the hive-c0re service:
1. The migration 'chown -R hive-core /var/lib/hyperhive' (which only
started running once ExecStartPre stopped failing 203/EXEC) stomped every
agent's bind-mounted creds — agents/<name>/{claude,state,harness,config}
are owned by the per-agent/manager users — logging all agents out with no
way back in. Scope it to everything *except* agents/, plus the agents/
dir node itself so c0re can still create new per-agent subdirs. Each
container's hive-agent-user-migrate activation chowns the contents back.
2. nix (prebuild 'nix build', flake-check, meta eval in c0re; nixos-container
update->nix in priv) writes its cache under $HOME/.cache. Both services
run as users with no home -> HOME=/var/empty (unwritable) -> Lix cache
init fails, rebuilds error out. Set HOME to each service's StateDirectory
(adding one for hive-priv).
hive-priv had no PATH, so the nixos-container it runs for every container
op (incl. lifecycle::list) failed ENOENT. build_all swallows that into an
empty list — the 'no managed containers' symptom. Give the helper the
minimal set nixos-container shells out to (nixos-container, nix, util-linux,
e2fsprogs) on top of the systemd/coreutils/findutils already in the unit
PATH, rather than the whole system profile.
Also route forge/matrix is_present() through priv_client::list_containers
instead of spawning nixos-container directly from unprivileged hive-c0re.
The state-migration chown invoked ${pkgs.coreutils}/bin/sh, which does not
exist (coreutils ships chown, not sh), so ExecStartPre exited 203/EXEC on
every boot. hive-c0re hit its start limit and never came up, so the gateway
returned 502 Bad Gateway. Call chown directly with the +- prefix (run as
root, tolerate failure) instead of going through a shell for '|| true'.
Per mara's direction: both nixpkgs and nixpkgs-unstable are now
top-level meta flake inputs with explicit store-path URLs. Hyperhive
follows them rather than the other way around:
inputs.nixpkgs.url = "path:${pkgs.path}";
inputs.nixpkgs-unstable.url = "path:${nixpkgs-unstable}";
inputs.hyperhive.url = "...";
inputs.hyperhive.inputs.nixpkgs.follows = "nixpkgs";
inputs.hyperhive.inputs.nixpkgs-unstable.follows = "nixpkgs-unstable";
New NixOS host options (auto-set at build time, overridable):
services.hyperhive.c0re.nixpkgsFlake
default: "path:${pkgs.path}" — host's evaluated nixpkgs.
services.hyperhive.c0re.nixpkgsUnstableFlake
default: "path:${nixpkgs-unstable}" from hyperhive's flake.nix —
the channel that carries claude-code. Operators can override to
track a different unstable snapshot.
Legacy fallback (both args empty) preserved for backward compat.
Two new Rust tests cover the full-URL and fallback paths.
