each queue card now shows its DAG's node chain (per-node state, step,
build-log link), fixing 'queue jumps don't show on the dashboard'.
live-log panel keys off the running node's log. new
services.hyperhive.c0re.buildSlots option (default 1) threads the
concurrent nix-build count into serve.json.
Agents in private netns can't reach host loopback directly; they
use forge.<domain> and matrix.<domain> via the gateway. Update four
locations that still claimed direct loopback access:
- hive-forge.nix: module header + openFirewall description
- hive-matrix.nix: openFirewall description
- docs/matrix.md: container design bullet + openFirewall section
Agents in private netns reach the forge at forge.<domain> via the
gateway, not via localhost directly. The conclusion (firewall only
matters for external access) is correct; the mechanism description
was stale from the pre-isolation era.
- Option doc: 'Default points at the in-host tuwunel (shared netns)'
→ updated to reflect that c0re forwards the gateway URL at runtime
- DNS oneshot comment: 'inert in shared-netns mode' → removed since
isolation is always on; oneshot always runs
Agents run in private netns (always-on isolation) and can't reach
host loopback directly. hive-c0re forwards HIVE_MATRIX_URL set to
`matrix.<domain>` via the gateway. Update two places that still
referenced the removed shared-netns path.
The option is a no-op (marked DEPRECATED — ignored); the Prerequisite,
Migration, and Rust counterpart sections all described the migration that
already completed. Strip to just the deprecation notice.
Also fix the bridgeIp description: 'once netns isolation lands' is past
tense — isolation is unconditional now. Rephrase to present tense.
privsep is complete and always on. the phased-rollout framing
is historical — drop it from the inline comments, keeping the
accurate description of what the user + delegation actually do.
claude --effort now accepts low, medium, high, xhigh, max.
Expand EFFORT_LEVELS from 3 to 5, update NixOS enum + description,
and fix the stale doc comment in web_ui.rs.
Closes#2220
The docs evaluation referenced self.nixosConfigurations.agent-base and
self.nixosModules.default — both tied to self's full flake store path.
Every commit produces a new self hash (even when only Rust or frontend
files change), so docs drvs change on every commit and the remote
builder (muede-pc2) must rebuild them from scratch each time.
Fix: use builtins.path on the nix/ directory with
builtins.unsafeDiscardStringContext to produce a content-addressed store
path (nixSrc) that only changes when .nix files change. Evaluate both
hostEval and agentEval from nixSrc using stub package args (pkgs.emptyFile /
pkgs.emptyDirectory) instead of self.overlays.default, which avoids
pulling in the Rust and frontend build closures.
Now docs drvs only change when a .nix file changes. PRs that touch only
Rust or frontend code hit the remote builder cache instead of forcing a
rebuild. Closes the disk-pressure-induced CI failure loop on muede-pc2.
Forge URLs in transformOptions updated to prepend nix/ prefix correctly
(nixSrc is a copy of nix/, so stripped paths need nix/ re-added).
hive-c0re never writes outside its StateDirectory + RuntimeDirectory:
- /etc writes (nixos-containers) go through hive-priv, not c0re
- /etc/hyperhive/serve.json is read-only at startup (written by the
NixOS activation script, not by the daemon)
- nix builds delegate to hive-priv via the privileged Unix socket;
hive-c0re itself never invokes nix directly
So strict is safe: systemd keeps /var/lib/hyperhive (StateDirectory)
and /run/hyperhive (RuntimeDirectory) writable; everything else is
read-only, shrinking the writable attack surface to exactly what the
service needs.
Root contexts (systemd services running as root, PID 1) default to
store=auto which resolves to the LOCAL nix store — bypassing the host
daemon, its remote builders (muede-pc2), and any prebuilt derivation
outputs already in the shared store. This causes spurious full
rebuilds of agent toplevels that the host already built and cached.
Two changes:
harness-base.nix:
- Add systemd.globalEnvironment.NIX_REMOTE = "daemon" — sets
DefaultEnvironment in systemd.conf so every unit in the container
inherits NIX_REMOTE=daemon. Non-root contexts already default to
the daemon socket; this only matters for root services that would
otherwise use the local store.
- Add NIX_REMOTE = "daemon" to environment.variables so interactive
shells also have it set (redundant with /etc/profile.d/nix-daemon.sh
but explicit and profile-agnostic).
hive-c0re.nix (hive-priv service):
- Add NIX_REMOTE = "daemon" to the service environment. hive-priv
runs as root and invokes nixos-container update + nix prebuild; these
must route through the host daemon so they see the shared store and
remote builders, not a private local store.
The sandbox-fallback = true in harness-base.nix is kept as a belt-
and-suspenders fallback but becomes a no-op for the common case once
nix routes through the daemon (the daemon builds on the host where
sandboxing works).
ProtectSystem=full makes /usr, /etc, and /boot read-only. Safe for
hive-c0re because it never writes to those paths directly:
- /etc writes (e.g. /etc/nixos-containers) go through hive-priv
- /etc/hyperhive/serve.json reads are read-only
- All runtime writes go to StateDirectory (/var/lib/hyperhive) and
RuntimeDirectory (/run/hyperhive), which remain writable under full
ProtectSystem=strict (everything read-only) is still deferred — it
requires auditing ReadWritePaths for every nix path c0re touches.
RestrictAddressFamilies deferred pending a watched deploy.
Also tightens the comment block to not mention a numbered issue.
hive-c0re runs as the unprivileged hive-core user and delegates all
privileged operations to hive-priv via a Unix socket. Add the subset of
systemd sandboxing directives that are provably safe without a watched
deploy (no ProtectSystem / RestrictAddressFamilies, which require mapping
the full set of filesystem/network paths first):
- NoNewPrivileges: already unprivileged; no setuid/capability escalation
- PrivateTmp: nix subprocesses use StateDirectory temp paths, not /tmp
- ProtectHome: HOME is /var/lib/hyperhive (StateDirectory), not /home/*
- ProtectKernelTunables: no sysctl writes needed
- ProtectKernelLogs: logs via systemd-journal group, not /dev/kmsg
- ProtectControlGroups: cgroup writes go through hive-priv, not c0re
- RestrictNamespaces: namespace creation goes through hive-priv
- LockPersonality: no personality changes needed
- RestrictRealtime: no real-time scheduling
Follows the same pattern as hive-priv (PrivateTmp + ProtectHome already
present there). Deferred: ProtectSystem=strict + ReadWritePaths +
RestrictAddressFamilies — those need a watched deploy per the #1845 plan
since getting the ReadWritePaths wrong silently breaks runtime I/O.
Previously, serveConfig was a pkgs.writeText store path embedded in
ExecStart. Any repo edit (gateway.nix, frontend, unrelated nix modules)
changes hyperhiveFlakeSource → serveConfig hash → ExecStart → systemd
restarts hive-c0re → StartupSweep → every agent rebuilt unnecessarily.
Fix: write the JSON to /etc/hyperhive/serve.json via environment.etc
(a managed /etc symlink to the store-path content). ExecStart now
references the stable /etc path, changing only when cfg.package (the
hive-c0re binary) changes. Unrelated module edits no longer bounce c0re
or trigger a cascade agent rebuild.
hive-c0re still restarts — and triggers the sweep — when its binary
changes, which is the correct invariant: a new release legitimately
needs to resync all agents with updated harness modules.
CLAUDE_CODE_OTEL_DIAG_STDERR was always set when OTEL is enabled, making
OTEL SDK errors (e.g. 404 from a misconfigured collector endpoint) appear
in every agent's stderr unconditionally. Move it behind a new opt-in flag.
Changes:
- nix/modules/hive-c0re.nix: add services.hyperhive.otel.debug (bool,
default false); wire to HYPERHIVE_OTEL_DEBUG env on hive-c0re unit.
- hive-c0re/src/meta.rs: add debug field to OtelConfig; read
HYPERHIVE_OTEL_DEBUG; emit hyperhive.otel.debug = true when set.
- nix/templates/harness-base.nix: add hyperhive.otel.debug internal
option; move CLAUDE_CODE_OTEL_DIAG_STDERR out of otelSettingsEnv into
a debug-gated lib.optionalAttrs block.
Default behaviour: OTEL exports silently (no stderr noise). Operators
troubleshooting collector connectivity set services.hyperhive.otel.debug
= true to re-enable the diagnostic output.
Add `baseClaudeEnv` (always-on) and extend `otelSettingsEnv` with the
env vars from #2149:
Always-on (every agent):
- CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
- CLAUDE_CODE_PLUGIN_PREFER_HTTPS=1
- CLAUDE_CODE_RESUME_INTERRUPTED_TURN=1 (recovers MCP-flap mid-turn)
- CLAUDE_CODE_SIMPLE_SYSTEM_PROMPT=1
- CLAUDE_CODE_SYNC_PLUGIN_INSTALL=1
- CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=hive+<agent>
- DISABLE_AUTOUPDATER=1 / DISABLE_UPDATES=1 (nix owns packages)
- DISABLE_INSTALL_GITHUB_APP_COMMAND=1
- DO_NOT_TRACK=1
- ENABLE_CLAUDEAI_MCP_SERVERS=0 (hive supplies its own)
- FORCE_AUTOUPDATE_PLUGINS=1
OTEL-gated (when hyperhive.otel.enable):
- CLAUDE_CODE_ENABLE_FEEDBACK_SURVEY_FOR_OTEL=1
- CLAUDE_CODE_OTEL_DIAG_STDERR=1
- OTEL_METRICS_INCLUDE_VERSION=1
Because baseClaudeEnv includes per-agent values
(CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX = "hive+${userName}"),
jq is now always run at build time — the OTEL-off branch that returned
the static asset verbatim is removed.
Per mara: the mirror seeding belongs in hive-c0re's forge provisioning
sweep, where the core admin token + org-ensure already live — not a parallel
host-side nix oneshot.
- forge.rs: ensure_mirrors() reads HYPERHIVE_FORGE_MIRRORS (JSON list of
{upstream,dest}), ensures each dest org (reuse ensure_org) + creates the
pull-mirror via the migrate API (reuse forge_http, serde_json::json! body,
409/existing = success). Called in ensure_all() right after the SEEDED_ORGS
loop (token in scope, warn-and-continue like the other ensure_* steps).
- hive-forge.nix: forward effectiveMirrors to c0re via
systemd.services.hive-c0re.environment.HYPERHIVE_FORGE_MIRRORS; drop the
forgejo-seed-mirrors.service + its script + the host-side core-token read.
Keep the forge.mirrors option, DEFAULT_ACTIONS_URL=self (CI-gated), and the
dest-shape / no-c0re-namespace-collision assertions.
Verified locally: nix parse + treefmt (incl rustfmt) clean; serde/serde_json
patterns mirror dashboard.rs. cargo build runs in CI (no cc in my container).
nixfmt wanted the ${...} on a single line (verified locally:
nix build .#checks.x86_64-linux.formatting passes). The earlier CI 'docs'
failure was a transient muede-pc2 build hiccup — the docs check builds clean
locally on the same drv.
- drop the three cross-ref tracker tags from comments/description (prose only)
- build the orgs + migrate JSON bodies with jq -n --arg (an upstream URL
containing a quote no longer corrupts the request)
- don't auto-append the actions/checkout mirror when the operator already
declared that dest (avoids a duplicate effectiveMirrors entry when CI is on)
General-purpose mirror mechanism for the internal forge, per mara's call on
#2074 (real Forgejo pull-mirrors, nix-configured — not a pushed clone).
- services.hyperhive.forge.mirrors: list of { upstream, dest } pull-mirrors,
any repo. Each is created as a real Forgejo pull-mirror (re-syncs from
upstream), dest = <owner>/<repo> in its own org.
- When forge.ci.enable is set: an actions/checkout mirror is auto-appended +
forgejo DEFAULT_ACTIONS_URL is pointed at this instance, so CI
'uses: actions/checkout@vN' resolves on loopback — immune to a host-resolver
blip that previously reded every checkout (the seed/re-sync needs external
DNS, but that's off the CI critical path).
- forgejo-seed-mirrors.service: host-side oneshot (the core admin token never
enters a container), modelled on hive-ci-prefetch — waits <=60s for the core
token, then idempotently ensures each dest org + creates the pull-mirror via
the migrate API. partOf the forge container so it re-ensures on restart.
- assertions: dest must be <owner>/<repo>; mirror orgs can't shadow the
c0re-managed namespaces (config/shared/agents/core) so the seed never races
hive-c0re's own provisioning.
Supersedes #2074 (the raw-clone stopgap) as the durable #2072 fix.
General-purpose Forgejo pull-mirror config (services.hyperhive.forge.mirrors:
list of {upstream, dest}). When CI is enabled, auto-append an actions/checkout
mirror + point forgejo DEFAULT_ACTIONS_URL at this instance so CI's
actions/checkout@vN resolves on loopback (immune to host-resolver blips, #2072).
Seed oneshot (creates the dest orgs + pull-mirrors via the migrate API) is the
next commit.
Per mara: a secret in the nix store is not acceptable. The non-secret
OTEL config (telemetry-enable, endpoint, protocol, resource attributes)
stays in the world-readable managed settings json; the auth header is
handled separately at runtime so it never touches the store.
New hive-otel-header oneshot (only when otel.enable && headersCredential
is set): inherits the forwarded otel-headers systemd credential via
LoadCredential, reads it at start, and merges OTEL_EXPORTER_OTLP_HEADERS
into the agent's 0600 ~/.claude/settings.json env block via jq. claude
layers the user env on top of the managed settings, so both the harness
turn-loop and hivectl choom (same agent user) export with auth. The
token is read from disk at start and never copied into the nix store or
the world-readable managed file.
Ordering is best-effort (before=, not a hard dep): a failure leaves the
harness running and telemetry exporting unauthenticated. headersCredential
option description updated to reflect it's now wired.
nix fmt clean.
The option description still claimed the credential is loaded via
systemd LoadCredential, but this PR removed that path. Clarify that the
option is currently inert (only the unauthenticated OTEL export is
implemented) and that runtime header injection is a planned follow-up,
so configuring it doesn't silently no-op without explanation.
Per mara: configure OTEL in the generated claude settings json (what the
Claude Code docs suggest), not a launch wrapper or /etc shell file.
claude-code auto-discovers /etc/claude-code/managed-settings.json in
every context — the harness turn loop AND hivectl choom — so putting the
OTEL env there gives telemetry parity declaratively, with no wrapper and
no --settings plumbing.
- managed-settings.json: was a static shared .source; now, when OTEL is
enabled, a per-agent build-time jq merge of the base asset + an env
block (jq at build, not eval-time readFile, to avoid IFD). OTEL off =
the static asset verbatim.
- otelSettingsEnv carries the static OTEL knobs + OTEL_RESOURCE_ATTRIBUTES
with the agent name (build-time) and the hive/swarm names forwarded by
meta.rs into environment.variables (mara: forward host config into
agent config where needed).
- removed the hive-serve-otel ExecStart wrapper, the per-unit otelEnv,
and the otel-headers LoadCredential from the harness service — the
harness binary emits no OTEL itself; only claude does, and it now reads
the settings json directly.
Known follow-ups (noted in code): the auth header (otel.headersCredential,
opt-in/default-null) is a secret and can't live in the world-readable
settings file — authenticated collectors need a runtime mechanism; this
PR covers the unauthenticated default.
nix fmt clean.
