`builtins.toJSON` already serialises a derivation as its out path and
`null` as `null`, so the `if … then null else "${…}"` binding was doing
by hand what the serialiser does anyway. Hand the package in whole and
drop the intermediate.
The gc-root property is unchanged and re-measured on the real module:
`environment.etc."hyperhive/serve.json".text` still `hasContext`, so the
host system closure still holds the package alive. Verified both ways —
with the package set the rendered JSON is byte-identical to the
interpolated version, and unset still emits `null` — with all module
assertions passing in each case.
The assertion now checks `builtins.toJSON cfg.claudeCodePackage`, which
is the value that actually has to carry the context, rather than an
intermediate that no longer exists.
mara on PR #2769: "make the default null instead of special casing """.
`claude_code_path` was a `String` whose empty value meant "no host-level
pin". That is a sentinel doing an `Option`'s job — the same shape argus
and mara already rejected on #2755's weights, and the same
empty-field cruft mara called out on #2756.
So it is `Option<String>` end to end:
- host module: `claudeCodePath` evaluates to `null` when
`claudeCodePackage` is unset, so `serve.json` carries JSON `null`
rather than `""`.
- `Coordinator` + `HiveEnv`: `Option<String>`, defaulting to `None`.
- `render_flake`/`render_flake_with_lookup`: `Option<&str>`, and the
emission is an `if let Some(path)` instead of an `is_empty()` guard.
- agent module: `hyperhive.claudeCodePath` is `nullOr str`, default
`null`.
Behaviour is unchanged in both directions; only the way "unset" is
spelled moves. The `builtins.hasContext` assertion still guards the
pinned case (short-circuited by the null check, so an unpinned hive
never evaluates it).
16/16 `meta::` tests, clippy clean, `nix fmt` no-op, `nix build .#docs`
green.
Agents run whatever `claude-code` the meta flake's `nixpkgs` resolves
to, and that is normally a release channel. This one package moves fast
enough that stable trails unstable by weeks — 26.05 is on 2.1.187 while
unstable carries 2.1.220 — and an agent cannot fix it for itself: it
only ever sees the single nixpkgs hive-c0re injects, so an `agent.nix`
has no other tree to reach for.
New host option `services.hyperhive.c0re.claudeCodePackage` takes the
package directly and rides the existing `hyperhiveDocs` threading path —
serveConfigJson -> HiveEnv -> render_flake — to reach each agent as
`hyperhive.claudeCodePath`. Null (the default) is today's behaviour.
What travels is the store *path*, as a plain string literal, not a flake
input: containers share the host's `/nix/store`, so the build is already
reachable inside them with its whole closure and has nothing to travel.
An input would be worse than useless — a `path:/nix/store/<pkg>` input
is re-copied as a reference-less `-source`, which strips exactly the
closure the binary needs.
The catch is that a path written into a generated flake is text, so
nothing in the container's closure keeps the binary alive. The host does
that instead, and gets it for free: the package is interpolated into
`/etc/hyperhive/serve.json`, `builtins.toJSON` preserves string context,
so the /etc entry references it and the system closure gc-roots it for
as long as that generation is the one the agents were rendered from. An
assertion pins that property, because losing the context is invisible at
eval and at deploy — it would surface only as every agent failing to
spawn `claude` whenever the next gc ran.
Container side wraps the path in a symlink farm rather than putting it
on PATH directly: `systemd.services.<name>.path` and
`environment.systemPackages` both coerce a store-path *string* through
`lib.toDerivation`, i.e. `builtins.storePath`, which pure evaluation
rejects. Interpolating the path into a builder is just text and
evaluates anywhere. `claude-code` drops out of systemPackages when a
pin is set, so there is exactly one claude in the container.
Refs #2693
Encoding "not configured" as weight 0 worked (the writer omitted the
line) but the type lied: 0 is not a legal cgroup v2 weight, and every
reader had to know the sentinel. Use Option<u32> end to end instead —
wire type, priv_client, HiveEnv, drop-in writer — so "unset" is a state
of the type rather than a magic value.
The nix options become nullOr, keeping their default of 80; null now
expresses "leave the setting out of the drop-in entirely" declaratively,
which is the useful shape on a host whose IO scheduler ignores
io.weight anyway.
Backward compat is unchanged: the fields stay #[serde(default)], so a
request from an older hive-c0re deserialises to None and reproduces the
pre-weights drop-in byte for byte. The test that pins that now passes
None instead of 0.
`CPUQuota=`/`MemoryMax=` are hard caps: they throttle an agent even when
the host is idle, so they are the wrong tool for "be polite under
contention". The cgroup v2 relative shares are, and neither was wired.
Adds `services.hyperhive.{agentCpuWeight,agentIoWeight}` (1..=10000,
default 80) threaded through the existing drop-in path: HiveEnv ->
write_dropins -> WriteResourceLimits -> hyperhive-limits.conf, next to
the caps already there. Hive-wide only, as the operator scoped it on the
issue: no per-agent override, no resource-limits.json field, no
dashboard form.
The default of 80 is below the kernel's 100, so agent containers yield
to everything *not* on this drop-in path -- host services and the infra
containers (hive-ci, hive-forge, hive-gateway, hive-matrix). It does not
rank agents against each other; they all carry the same weight.
`WriteResourceLimits` gains two `#[serde(default)]` fields, and the
writer treats weight 0 as "not configured" and omits the line, so an
older hive-c0re talking to a newer hive-priv still produces the exact
pre-weights drop-in. The body is extracted into `limits_dropin_body` so
that is covered by a test rather than asserted by eye.
The gateway container's /etc/resolv.conf is a one-shot copy: nixos-container
cps it in from the host in its start script, and nspawn's --resolv-conf=auto
copies (not binds) for a writable host-netns container. systemd-nspawn(1)
states the consequence outright — "no further propagation of configuration is
generally done after the one-time early initialization (this is because the
file is usually updated through copying and renaming)".
dnsmasq has no explicit upstream and follows that file, so a host network
change strands it on a resolver that no longer answers and every non-hive
lookup from every agent hangs. Agents' own resolvers point at the static
bridge IP and never go stale, which is why the symptom presents as "the
gateway needs a kick".
Add a host-side hive-gateway-resolv path unit watching /etc/resolv.conf.
On change it machinectl copy-to's the file into the container and reloads
dnsmasq — ExecReload is kill -HUP, so upstreams are re-read and the cache
flushed without dropping anything; nginx never notices.
- watched from the HOST: a rename on the host doesn't cross the nspawn mount
namespace, so an in-container path unit can't see it (same reason c0re
reloads nginx from the host side)
- copy, not a file bind-mount: openresolv renames over the file, so a bind
would pin the first inode forever — strictly worse than today
- machinectl copy-to writes through the container's own mount namespace, so
this holds regardless of how the container assembles /etc
- armed Before=network-pre.target so the boot's first DHCP write is caught,
and re-run on gateway start for changes made while it was down
- a host file with no nameserver line is skipped, not pushed, so a
mid-rewrite snapshot can't blank hive DNS
- deliberately no fallback server=: dnsmasq queries all known upstreams in
parallel, so a hardcoded public resolver would take a share of normal
traffic rather than only covering the gap
Container nix invocations hard-failed whenever the remote builder
muede-pc2 was unreachable, while the identical build on the host
succeeded. Both go through the same host nix-daemon, so the difference
looked impossible.
The cause is that `fallback` is a client-side option: the nix client
transmits it to the daemon per connection (`tryFallback` in setOptions),
so the caller decides whether a failed remote dispatch may degrade to a
local build — even when the build itself runs on the host daemon under
NIX_REMOTE=daemon. Only genuinely daemon-side settings such as
`builders` are inherited from the host. The containers never set
`fallback`, so they took nix's default of false.
Set it in the agent-container base module and in the CI container, and
correct the hive-ci comment that claimed fallback was inherited from
the host daemon along with buildMachines and max-jobs.
Verified in an agent container: `nix fmt` fails outright on the remote
store's connection reset, while the same command with fallback enabled
reports the same connection error and then builds locally and succeeds.
The host admin socket `/run/hyperhive/host.sock` was `0660 root:root` (no
SocketGroup), so hivectl needed sudo. Group-own it by a new `hive-admin`
group and add a `services.hyperhive.c0re.adminUsers` allowlist: listed users
join `hive-admin` and drive hivectl without root.
- `SocketGroup = "hive-admin"`, `SocketMode = "0660"` on the hive-c0re.socket
unit.
- `/run/hyperhive` -> `0751` (traverse-only, no listing) so the group can reach
the socket path; the socket's own `0660 hive-admin` mode gates the
connection, and the per-agent subdirs keep their own restrictive perms.
- Empty `adminUsers` (the default) leaves `hive-admin` memberless -> root-only,
as before.
The admin socket is full hive control (spawn/kill/destroy/deploy), so
`adminUsers` is an explicit, opt-in trust grant. Documented in
docs/boundary.md (host admin socket access) + docs/tools/hivectl.md.
CI's nix flake check ran in hive-ci's OWN in-container nix-daemon, which
offloads to the pc2 remote builder and HARD-FAILS when pc2 is
unreachable (Connection-reset) — reddening every PR's queue hive-wide.
The host daemon builds fine in the same situation (buildMachines +
max-jobs>=1 + fallback → local build when pc2 is down), and the agent
containers already route through it.
Give hive-ci the same wiring: bind-mount the host nix-daemon socket dir
into the container, set NIX_REMOTE=daemon, and disable the container's
own nix-daemon service + socket. Now CI builds through the host daemon —
pc2 offload when it's up, graceful local fallback when it's down. Drops
the now-moot in-container wait-nix-daemon precond. Needs an operator
rebuild to apply.
hive-core now fetches each agent's config as a forge-hosted flake input
(git+http://<forge>/agent-configs/<n>.git). Add a git credential helper
(git-credential-hive-forge) that reads the live forge-core admin token on
every invocation and authenticates as the forge core user, wired via the
[credential] stanza in hive-core's $HOME/.gitconfig + on the service PATH.
Reading the token file live means zero stale copies and no resync on
rotation; no token ever lands in a flake URL or lock. Mirrors the existing
github.nix credential-helper pattern.
Completes #2415. Registration no longer gates container@hive-ci start:
- Retire the host-side hive-ci-prefetch.service (+ its 100-line script and
the now-dead coreTokenPath/containerRoot let-bindings) — the forge round-trip
it did on the boot path now lives in hive-c0re (forge/ci_runner.rs), run out
of band during the startup sweep.
- Drop the container@hive-ci TimeoutStartSec = mkForce 180s band-aid that
papered over that boot-path wait tripping the nspawn start timeout (the ~60s
restart loop of #2410).
- gitea-runner-hive gains an ExecStartPre precond (ahead of the nix-daemon
wait) that fails fast unless it is already registered (.runner present) or a
real, non-placeholder token is in place — so missing creds just hold the
runner down instead of blocking the container. Restart=on-failure (no start
limit) self-heals it once hive-c0re writes the token; c0re's explicit restart
is the primary path.
- Set HYPERHIVE_FORGE_CI_ENABLED=1 on hive-c0re.service so the sweep registers.
The tmpfiles TOKEN=placeholder seed + read-only bind-mount are unchanged; the
core admin token still never enters the container.
hive-c0re already samples each agent container's cgroup load for the
dashboard (stats/container_stats.rs); this rides those gauges out to the
configured OTLP endpoint, reusing the existing services.hyperhive.otel
config (endpoint + auth header) — no new toggle.
- New stats/otel_metrics.rs: exports via the OpenTelemetry Rust SDK (same
crates as hive-metric) with the semconv container.* metric names +
container.name attribute so off-the-shelf OTel/Grafana dashboards work,
plus the hive agent label. container.cpu.time (counter, s, from cumulative
cpu.stat usage_usec), container.memory.usage, container.memory.usage.limit;
memory peak / on-disk storage / instantaneous cpu percent stay hyperhive.*
custom (no semconv equivalent). Observable instruments read a shared
snapshot an async task refreshes (gather() is async; SDK callbacks sync).
- container_stats: expose cpu_time_usec (cumulative) on ContainerResource.
- The OTLP auth header is loaded onto hive-c0re's own unit via systemd
LoadCredential and read from $CREDENTIALS_DIRECTORY/otel-headers.
- docs/observability.md documents the host-emitted semconv metrics.
Host-side export, so it covers containers even when their agent is idle.
hive-ci and hive-forge both bind the runtime-generated hive CA cert
read-only and order their container@ unit after hive-tls-ca.service so
the bind source exists before nspawn sets the mount up — the same
bind-mount + ordering + rationale duplicated verbatim in two modules.
Extract that language-agnostic half into a pure helper,
nix/host-modules/lib/hive-ca-trust.nix, taking a container name and
returning { useSelfSigned, caContainerPath, bindMount, containerOrdering }.
The per-runtime consumption stays at each call site (hive-ci's additive
NODE_EXTRA_CA_CERTS, hive-forge's Go SSL_CERT_FILE concat). hive-ci folds
containerOrdering into its existing mkMerge alongside the TimeoutStartSec
bump.
The helper is a pure function, not a module: host-modules/default.nix is
an explicit aggregator (not a glob) and the docs eval imports that same
aggregator, so the lib/ file is never picked up as a module. A third
outbound-TLS-trusting container no longer means a third copy-paste.
Per mara's feedback on PR #2407 ("better: you can also provide url in
dashboard, same as with matrix, no host config"), drops
services.hyperhive.extraForges and the admin-API mint/revoke flow
entirely. The operator now creates a token on the external forge
themselves and pastes a label + base URL + access token into the
dashboard's FORGES tab, the same shape as the GitHub PAT flow plus the
base-URL field from the matrix extra-account flow. hive-c0re only ever
writes/deletes two local files per account (forge-<label>-token,
forge-<label>.json sidecar for the URL) via hive-priv — no remote
account creation, no admin token, no revoke-on-the-remote-side, no nix
config to enumerate.
- nix/host-modules/hive-forge/default.nix: removed the extraForges
option, its label-format assertion, and the HYPERHIVE_EXTRA_FORGES
env forwarding.
- hive-c0re/src/forge/extra.rs: deleted (REST admin-API provisioning,
no longer needed).
- hive-c0re/src/dashboard/extra_forges.rs: GET /api/extra-forges?
agent= lists an agent's stored forges by scanning its state dir
(mirrors matrix_accounts.rs's filename-scan listing), POST
/api/extra-forge-account (agent/label/base_url/token/
action=add|remove) stores or removes an account.
- hive-sh4re/priv_proto.rs + hive-priv/main.rs: new
WriteAgentExtraForgeAccount/DeleteAgentExtraForgeAccount priv
requests (adds base_url, writes/deletes a JSON sidecar alongside the
token).
- hive-c0re/src/priv_client.rs: matching wrapper functions.
- frontend/packages/dashboard/src/credentials.{html,js}: FORGES tab is
a per-agent list + add-account paste form (label/base_url/token), no
grant/revoke-from-catalog UI.
- docs/web-ui/dashboard.md: FORGES tab section rewritten.
Supersedes the design in PR #2407 (already approved+green on the old
admin-API model) — opening as a fresh PR against the same issues
rather than force-pushing over the approved one.
gitea-runner registration (hive-ci-prefetch, host-side) sits on the
boot-critical path -- nspawn readiness is gated on the runner
registering, itself a forge + core-token round trip that can wait up
to 60s for the core token. The default ~60s TimeoutStartSec can trip
mid-register (especially right after a .runner purge, since every
boot re-registers from scratch), killing the half-started container
and triggering a restart loop until the token/forge settle. Bump to
180s so one register attempt has room to finish.
