On a fresh deploy Persistent=true causes the weekly timer to fire
immediately. Without this ordering hive-tls-resign can race
hive-tls-ca before the CA key is written, failing the resign.
CA is always valid when this runs; resign is a no-op if the leaf
is still within its half-life margin.
Suggested by argus (review) + requested by mara.
The parked PR (#1934) triggered re-sign by restarting hive-tls-ca
(a RemainAfterExit oneshot), which mara flagged as a hack. This
replaces that with a dedicated hive-tls-resign.{service,timer}:
- hive-tls-ca unchanged — still handles boot-time CA generation +
initial leaf signing; RemainAfterExit, runs before the gateway.
- hive-tls-resign.service — runs the leaf re-sign directly (openssl
in its own PATH); does NOT touch hive-tls-ca. Re-signs only when
the leaf is within half its validity of expiry, so the weekly timer
catches short-lived (30d default) leaves well before they lapse.
Checks sha256 before/after; if the leaf rotated it propagates into
the running gateway via systemctl -M hive-gateway (mirroring how
hive-c0re reloads the gateway after agents.conf writes — the
documented correct approach; an inotify path unit inside the
container does not work across the nspawn mount-namespace boundary).
|| true on the propagation calls so a stopped gateway never fails
the unit (its next boot imports the already-rotated leaf).
- hive-tls-resign.timer — OnCalendar=weekly, Persistent=true (fires a
missed run on next boot so a host that was off on the scheduled day
catches up rather than letting the leaf lapse silently).
- leafValidityDays doc updated to reflect automatic renewal.
Closes#1933.
hive-ci is the one container on the hive bridge that used DHCP
(networking.interfaces.eth0.useDHCP = true, from #2310 / #2336). But the
bridge has no DHCP server: dnsmasq on the bridge is DNS-only and agent
containers get deterministic static IPs (lifecycle::agent_network_ip),
so the DHCP client never gets a lease and the container hangs at boot:
A start job is running for DHCP Client (5s / 1min 31s)
Assign a static address on eth0 (top host address of the subnet) plus a
default route via the bridge gateway, and drop useDHCP. Preserves the
#2310 netns isolation (no host-loopback reach) while letting the
container boot and reach the forge through the gateway.
Refs #2310, #2336.
nixos-containers.nix sets networking.useDHCP = false for every
container (to suppress the deprecated global DHCP warning). Setting
networking.useDHCP = true in the hive-ci inner config conflicts with
no priority differential → eval error on hosts running current main.
Fix: drop the global useDHCP = true and configure DHCP on eth0
(the inner veth interface name assigned by systemd-nspawn when
hostBridge is used). This is both conflict-free and the correct
modern NixOS practice for per-interface DHCP.
The container-level nix.settings.fallback = lib.mkForce true (added as an
interim safety net so a down remote builder degrades gracefully rather than
hard-failing CI) is rendered redundant by the host-daemon routing fix: with
NIX_REMOTE=daemon set for all systemd units in the container, every nix
build invocation routes through the host daemon, which already has its own
fallback policy. The container setting would only apply to nix invocations
that bypass the daemon entirely — a case that should not exist post-deploy.
NOTE: draft — gated on the daemon-routing change (PR that adds
NIX_REMOTE=daemon to systemd.globalEnvironment + hive-priv service env)
being deployed and confirmed working in production before merging.
Give hive-ci its own network namespace (privateNetwork=true, hostBridge)
instead of sharing the host netns. This closes the path by which a
prompt-injected PR nix build could reach the unauthenticated dashboard
at 127.0.0.1:7000.
The runner now reaches the forge through the gateway:
- networking.extraHosts maps forgeCfg.domain → bridgeIp
- runner url = http://${forgeCfg.domain} (gateway port 80, addSSL=true,
no HTTP→HTTPS redirect — HTTP is served alongside HTTPS)
- networking.useDHCP = true for the bridge-attached veth
- networking.nameservers = [bridgeIp] (dnsmasq on bridge, external fwd)
Ports 80 and 443 are already open on the bridge firewall (hive-network.nix).
The bridge→127.0.0.0/8 DROP rule (defence-in-depth) is unchanged — CI
can no longer reach 127.0.0.1:{7000 dashboard, forge httpPort, etc.}.
Requires forge.behindGateway=true (the default when hyperhive is enabled)
— asserted so misconfigured hosts get a clear build error.
Note: the host-side hive-ci-prefetch script still uses
127.0.0.1:${httpPort} directly (it runs on the host, not in the
container) — no change needed there.
Replace the hardcoded FORGE_HTTP const with forge_http_base() which
reads HIVE_FORGE_URL from the environment (already set unconditionally
by hive-c0re.nix to http://<forge.domain>). Add forge_git_url() helper
that inserts core:<token> credentials between scheme and authority for
git push/clone URLs.
All call sites updated:
- forge/mod.rs: api() OnceLock + new forge_git_url/forge_http_base fns
- forge/repos.rs: push_meta, push_config, ensure_meta_remote
- forge/pr_merge.rs: tokenised_repo_url delegate + test loosened
- workers/knowledge.rs: clone + push URLs
- socket_server/mod.rs: clone_url in RepoCreated response
No new env var: HIVE_FORGE_URL was already the right knob (mara).
Closes#1868. Closes#2174 (this supersedes the operators-team fix from
the closed#2218, which is re-applied in the ensure_operators_team call
that was already merged separately).
Adds hive-xdg-icons package (nix/packages/hive-xdg-icons.nix) that
rasterizes the branding SVG to all standard hicolor sizes (16–256px)
and installs /usr/share/pixmaps/hyperhive.png as a flat fallback.
Ships .desktop entries for hive-c0re and hive-ag3nt with NoDisplay=true
so desktop environments can match running processes to their icon without
cluttering the application launcher.
Narrow drv input: only ./branding/hyperhive.svg, so unrelated source
changes don't bust the cache.
Wired into environment.systemPackages in hive-c0re.nix (host side)
so the icons are present wherever hive-c0re is deployed.
Closes#2228
each queue card now shows its DAG's node chain (per-node state, step,
build-log link), fixing 'queue jumps don't show on the dashboard'.
live-log panel keys off the running node's log. new
services.hyperhive.c0re.buildSlots option (default 1) threads the
concurrent nix-build count into serve.json.
Agents in private netns can't reach host loopback directly; they
use forge.<domain> and matrix.<domain> via the gateway. Update four
locations that still claimed direct loopback access:
- hive-forge.nix: module header + openFirewall description
- hive-matrix.nix: openFirewall description
- docs/matrix.md: container design bullet + openFirewall section
Agents in private netns reach the forge at forge.<domain> via the
gateway, not via localhost directly. The conclusion (firewall only
matters for external access) is correct; the mechanism description
was stale from the pre-isolation era.
The option is a no-op (marked DEPRECATED — ignored); the Prerequisite,
Migration, and Rust counterpart sections all described the migration that
already completed. Strip to just the deprecation notice.
Also fix the bridgeIp description: 'once netns isolation lands' is past
tense — isolation is unconditional now. Rephrase to present tense.
privsep is complete and always on. the phased-rollout framing
is historical — drop it from the inline comments, keeping the
accurate description of what the user + delegation actually do.
hive-c0re never writes outside its StateDirectory + RuntimeDirectory:
- /etc writes (nixos-containers) go through hive-priv, not c0re
- /etc/hyperhive/serve.json is read-only at startup (written by the
NixOS activation script, not by the daemon)
- nix builds delegate to hive-priv via the privileged Unix socket;
hive-c0re itself never invokes nix directly
So strict is safe: systemd keeps /var/lib/hyperhive (StateDirectory)
and /run/hyperhive (RuntimeDirectory) writable; everything else is
read-only, shrinking the writable attack surface to exactly what the
service needs.
Root contexts (systemd services running as root, PID 1) default to
store=auto which resolves to the LOCAL nix store — bypassing the host
daemon, its remote builders (muede-pc2), and any prebuilt derivation
outputs already in the shared store. This causes spurious full
rebuilds of agent toplevels that the host already built and cached.
Two changes:
harness-base.nix:
- Add systemd.globalEnvironment.NIX_REMOTE = "daemon" — sets
DefaultEnvironment in systemd.conf so every unit in the container
inherits NIX_REMOTE=daemon. Non-root contexts already default to
the daemon socket; this only matters for root services that would
otherwise use the local store.
- Add NIX_REMOTE = "daemon" to environment.variables so interactive
shells also have it set (redundant with /etc/profile.d/nix-daemon.sh
but explicit and profile-agnostic).
hive-c0re.nix (hive-priv service):
- Add NIX_REMOTE = "daemon" to the service environment. hive-priv
runs as root and invokes nixos-container update + nix prebuild; these
must route through the host daemon so they see the shared store and
remote builders, not a private local store.
The sandbox-fallback = true in harness-base.nix is kept as a belt-
and-suspenders fallback but becomes a no-op for the common case once
nix routes through the daemon (the daemon builds on the host where
sandboxing works).
ProtectSystem=full makes /usr, /etc, and /boot read-only. Safe for
hive-c0re because it never writes to those paths directly:
- /etc writes (e.g. /etc/nixos-containers) go through hive-priv
- /etc/hyperhive/serve.json reads are read-only
- All runtime writes go to StateDirectory (/var/lib/hyperhive) and
RuntimeDirectory (/run/hyperhive), which remain writable under full
ProtectSystem=strict (everything read-only) is still deferred — it
requires auditing ReadWritePaths for every nix path c0re touches.
RestrictAddressFamilies deferred pending a watched deploy.
Also tightens the comment block to not mention a numbered issue.
hive-c0re runs as the unprivileged hive-core user and delegates all
privileged operations to hive-priv via a Unix socket. Add the subset of
systemd sandboxing directives that are provably safe without a watched
deploy (no ProtectSystem / RestrictAddressFamilies, which require mapping
the full set of filesystem/network paths first):
- NoNewPrivileges: already unprivileged; no setuid/capability escalation
- PrivateTmp: nix subprocesses use StateDirectory temp paths, not /tmp
- ProtectHome: HOME is /var/lib/hyperhive (StateDirectory), not /home/*
- ProtectKernelTunables: no sysctl writes needed
- ProtectKernelLogs: logs via systemd-journal group, not /dev/kmsg
- ProtectControlGroups: cgroup writes go through hive-priv, not c0re
- RestrictNamespaces: namespace creation goes through hive-priv
- LockPersonality: no personality changes needed
- RestrictRealtime: no real-time scheduling
Follows the same pattern as hive-priv (PrivateTmp + ProtectHome already
present there). Deferred: ProtectSystem=strict + ReadWritePaths +
RestrictAddressFamilies — those need a watched deploy per the #1845 plan
since getting the ReadWritePaths wrong silently breaks runtime I/O.
Previously, serveConfig was a pkgs.writeText store path embedded in
ExecStart. Any repo edit (gateway.nix, frontend, unrelated nix modules)
changes hyperhiveFlakeSource → serveConfig hash → ExecStart → systemd
restarts hive-c0re → StartupSweep → every agent rebuilt unnecessarily.
Fix: write the JSON to /etc/hyperhive/serve.json via environment.etc
(a managed /etc symlink to the store-path content). ExecStart now
references the stable /etc path, changing only when cfg.package (the
hive-c0re binary) changes. Unrelated module edits no longer bounce c0re
or trigger a cascade agent rebuild.
hive-c0re still restarts — and triggers the sweep — when its binary
changes, which is the correct invariant: a new release legitimately
needs to resync all agents with updated harness modules.
CLAUDE_CODE_OTEL_DIAG_STDERR was always set when OTEL is enabled, making
OTEL SDK errors (e.g. 404 from a misconfigured collector endpoint) appear
in every agent's stderr unconditionally. Move it behind a new opt-in flag.
Changes:
- nix/modules/hive-c0re.nix: add services.hyperhive.otel.debug (bool,
default false); wire to HYPERHIVE_OTEL_DEBUG env on hive-c0re unit.
- hive-c0re/src/meta.rs: add debug field to OtelConfig; read
HYPERHIVE_OTEL_DEBUG; emit hyperhive.otel.debug = true when set.
- nix/templates/harness-base.nix: add hyperhive.otel.debug internal
option; move CLAUDE_CODE_OTEL_DIAG_STDERR out of otelSettingsEnv into
a debug-gated lib.optionalAttrs block.
Default behaviour: OTEL exports silently (no stderr noise). Operators
troubleshooting collector connectivity set services.hyperhive.otel.debug
= true to re-enable the diagnostic output.
Per mara: the mirror seeding belongs in hive-c0re's forge provisioning
sweep, where the core admin token + org-ensure already live — not a parallel
host-side nix oneshot.
- forge.rs: ensure_mirrors() reads HYPERHIVE_FORGE_MIRRORS (JSON list of
{upstream,dest}), ensures each dest org (reuse ensure_org) + creates the
pull-mirror via the migrate API (reuse forge_http, serde_json::json! body,
409/existing = success). Called in ensure_all() right after the SEEDED_ORGS
loop (token in scope, warn-and-continue like the other ensure_* steps).
- hive-forge.nix: forward effectiveMirrors to c0re via
systemd.services.hive-c0re.environment.HYPERHIVE_FORGE_MIRRORS; drop the
forgejo-seed-mirrors.service + its script + the host-side core-token read.
Keep the forge.mirrors option, DEFAULT_ACTIONS_URL=self (CI-gated), and the
dest-shape / no-c0re-namespace-collision assertions.
Verified locally: nix parse + treefmt (incl rustfmt) clean; serde/serde_json
patterns mirror dashboard.rs. cargo build runs in CI (no cc in my container).
nixfmt wanted the ${...} on a single line (verified locally:
nix build .#checks.x86_64-linux.formatting passes). The earlier CI 'docs'
failure was a transient muede-pc2 build hiccup — the docs check builds clean
locally on the same drv.
- drop the three cross-ref tracker tags from comments/description (prose only)
- build the orgs + migrate JSON bodies with jq -n --arg (an upstream URL
containing a quote no longer corrupts the request)
- don't auto-append the actions/checkout mirror when the operator already
declared that dest (avoids a duplicate effectiveMirrors entry when CI is on)
General-purpose mirror mechanism for the internal forge, per mara's call on
#2074 (real Forgejo pull-mirrors, nix-configured — not a pushed clone).
- services.hyperhive.forge.mirrors: list of { upstream, dest } pull-mirrors,
any repo. Each is created as a real Forgejo pull-mirror (re-syncs from
upstream), dest = <owner>/<repo> in its own org.
- When forge.ci.enable is set: an actions/checkout mirror is auto-appended +
forgejo DEFAULT_ACTIONS_URL is pointed at this instance, so CI
'uses: actions/checkout@vN' resolves on loopback — immune to a host-resolver
blip that previously reded every checkout (the seed/re-sync needs external
DNS, but that's off the CI critical path).
- forgejo-seed-mirrors.service: host-side oneshot (the core admin token never
enters a container), modelled on hive-ci-prefetch — waits <=60s for the core
token, then idempotently ensures each dest org + creates the pull-mirror via
the migrate API. partOf the forge container so it re-ensures on restart.
- assertions: dest must be <owner>/<repo>; mirror orgs can't shadow the
c0re-managed namespaces (config/shared/agents/core) so the seed never races
hive-c0re's own provisioning.
Supersedes #2074 (the raw-clone stopgap) as the durable #2072 fix.
General-purpose Forgejo pull-mirror config (services.hyperhive.forge.mirrors:
list of {upstream, dest}). When CI is enabled, auto-append an actions/checkout
mirror + point forgejo DEFAULT_ACTIONS_URL at this instance so CI's
actions/checkout@vN resolves on loopback (immune to host-resolver blips, #2072).
Seed oneshot (creates the dest orgs + pull-mirrors via the migrate API) is the
next commit.
Follow-up to the harness fix: any unit whose ExecStart is a generated
wrapper (a NixOS `script =` block or a writeShellScript) gets its journal
SyslogIdentifier derived from the wrapper's store-path basename — an
opaque, unstable name. Pin an explicit stable identifier on every unit
hyperhive declares.
script/wrapper units (leaked store paths): forgejo-gpg-init,
hive-gateway-self-signed-cert, hive-tls-ca, hive-ci-prefetch,
hyperhive-isolated-dns, tea-login, forge-avatar-sync,
matrix-avatar-sync, hive-dashboard-links.
direct-binary units (basename already sane; explicit for tidiness +
future-proofing): hive-c0re, hive-priv, hive-matrix-daemon,
hive-bash-daemon, weston.
Only changes the displayed line identifier; journalctl -u <unit>
filtering (keys on _SYSTEMD_UNIT) is unaffected.
Short-lived leaf bounds the blast radius of a key compromise. Doc notes
the renewal caveat: re-sign is activation-only today, so a hive that
stays up >30 days without a rebuild would need a periodic re-sign timer
to avoid leaf expiry (follow-up).
A 10-year leaf is pointless given the leaf auto-rotates (re-signed
within 30 days of expiry) and is long enough to make browsers warn.
397 days sits just under the 398-day CA/Browser-Forum leaf cap (so
browsers accept it) while bounding the blast radius of a leaf-key
compromise. The re-sign runs at service activation (hive-tls-ca is a
oneshot, no timer), so 397 stays comfortably above any gateway
rebuild gap.
The dashboard vhost split static-vs-backend on the request Accept
header (map $http_accept $dashboard_spa_target), so the same URL
behaved differently by content-type — e.g. /api/state fetched with
Accept: text/html wrongly returned index.html.
Now that all hive-c0re routes live under /api/ plus the single
/webhook/knowledge endpoint, route by path instead: /api/ and
/webhook/ proxy to c0re (SSE settings on /api/), everything else
serves the dist with try_files $uri /index.html. Drops the
dashboard Accept-header map and the @c0re named location.
Updates docs/gateway.md accordingly.
`nixos-container update <agent>` (run by the hive-priv root helper) invokes
nix, which shells out by bare name to `git` whenever it has to fetch or
re-resolve a git-source flake input, and to `ssh` to dispatch builds to
remote builders. The hive-priv service `path` had neither, so an agent whose
config carries a git-source input — or a stale flake.lock whose node URL no
longer matches the flake's declared input, forcing nix to re-resolve at eval
— fails the build with `executing "git": No such file or directory`, and
remote-builder dispatch fails with `Could not find executable 'ssh'`.
Add `pkgs.gitMinimal` + `pkgs.openssh` to the path. Latent-bug fix: nix
legitimately needs both for any git-source input / remote build. Surfaced by
the bitburner agent's build failing on a git-source input re-resolution.
The internal forge is the canonical store for the meta flake, every
agent's config repo, and the internal/* repos, so it can no longer be
optional. Remove the services.hyperhive.forge.enable toggle:
- hive-forge.nix: drop the `enable` option; the forge config now
deploys gated on `services.hyperhive.enable` (it ships with hyperhive).
- hive-c0re.nix: HIVE_FORGE_URL env unconditional; forge-public-URL gate
drops the enable check (keeps behindGateway).
- hive-gateway.nix: local /etc/hosts forge entry keyed on behindGateway.
- hive-ci.nix: drop the now-moot `forge.ci.enable requires forge.enable`
assertion (forge is always present); reword the option doc.
- nix/docs/default.nix: drop the `forge.enable = mkForce false` stub
(option gone); the options-doc eval stays light via hyperhive.enable.
- hive-c0re forge.rs / hivectl.rs: reword 'forge.enable = true' error
text to 'wait for hive-c0re to start the container' (the runtime
token-absent path is unchanged — it's a bootstrap-timing check, not
the opt-out).
- docs/approvals.md, docs/ci.md: drop stale forge.enable references.
Migration: configs that set `services.hyperhive.forge.enable = false`
must drop the line — the forge is now mandatory.
Prereq/companion to #1838 (PR-based config flow, which assumes the forge
is always present).
Add swarm.peers.<domain>.caCert (path to a peer hive's root CA PEM),
trusted everywhere the hive's own internal CA is — so a self-signed
peer hive can federate (matrix) and any in-hive consumer validates its
certs.
Mechanism (reuses the existing hive-CA embedding): the meta-flake
renderer embeds a LIST of CA files next to each agent's flake —
hive-ca.pem (the hive's own self-signed CA, when active) plus each peer
caCert as peer-ca-<N>.pem — and emits them all in
security.pki.certificateFiles, so every agent trusts them at build
time. The matrix container trusts the same peer CAs for federation TLS.
Nothing is installed in the host trust store; the certs live in the nix
store (no mutable host file).
- meta.rs: embedded_ca_files() = hive CA + peer CAs (from new
HIVE_PEER_CA_PATHS env); ca_embed_state() tracks the list (content +
add/remove); sync_agents materialises + stages the list; render emits
the multi-entry certificateFiles. Tests cover hive-only / hive+peers
/ peers-only / none.
- hive-c0re.nix: HIVE_PEER_CA_PATHS service env (colon-joined caCerts);
caCert / certFingerprint option docs updated to the hive-wide scope.
- hive-matrix.nix + docs/swarm.md: scope + comment updates.
certFingerprint stays the c0re-only leaf-pin path.
