The internal forge is the canonical store for the meta flake, every
agent's config repo, and the internal/* repos, so it can no longer be
optional. Remove the services.hyperhive.forge.enable toggle:
- hive-forge.nix: drop the `enable` option; the forge config now
deploys gated on `services.hyperhive.enable` (it ships with hyperhive).
- hive-c0re.nix: HIVE_FORGE_URL env unconditional; forge-public-URL gate
drops the enable check (keeps behindGateway).
- hive-gateway.nix: local /etc/hosts forge entry keyed on behindGateway.
- hive-ci.nix: drop the now-moot `forge.ci.enable requires forge.enable`
assertion (forge is always present); reword the option doc.
- nix/docs/default.nix: drop the `forge.enable = mkForce false` stub
(option gone); the options-doc eval stays light via hyperhive.enable.
- hive-c0re forge.rs / hivectl.rs: reword 'forge.enable = true' error
text to 'wait for hive-c0re to start the container' (the runtime
token-absent path is unchanged — it's a bootstrap-timing check, not
the opt-out).
- docs/approvals.md, docs/ci.md: drop stale forge.enable references.
Migration: configs that set `services.hyperhive.forge.enable = false`
must drop the line — the forge is now mandatory.
Prereq/companion to #1838 (PR-based config flow, which assumes the forge
is always present).
The CI container set `nix.settings.sandbox-fallback` but never
`nix.settings.fallback`, so a cache-miss build with an unreachable
remote build machine had no local fallback and hard-failed. A single
down or DNS-unresolvable `nix.buildMachines` host therefore turned every
fresh `nix flake check` red hive-wide, in ~30s, even for diffs that
can't affect the nix build (pure JS/markdown). Add
`nix.settings.fallback = true` so CI degrades to a slower local build
instead of failing.
A bare `nix store ping` run as root with the daemon socket absent
resolves to a local store (root writes /nix/store directly) and pings
successfully, so the gate could false-pass at cold boot: nix-daemon.socket
carries ConditionPathIsReadWrite=/nix/var/nix/daemon-socket and is
condition-skipped until /nix/var goes read-write. systemd service units
don't source the profile that sets NIX_REMOTE=daemon, so auto-store
fallback to local is the real environment here.
Pin --store daemon so the readiness poll verifies the actual daemon
socket and the gate honours its wait-for-daemon contract instead of
passing against a local fallback while the daemon is still down.
Per review: the `tls.certDir == null && !tls.acme.enable` derivation was
duplicated in hive-gateway, hive-tls, and hive-ci. Expose it once as a
read-only internal option `services.hyperhive.gateway.useSelfSigned` (the
gateway module's single source of truth) and have hive-tls and hive-ci
consume it instead of re-deriving.
Eval-proven: gateway.useSelfSigned is true on the self-signed default /
false with tls.certDir, and the hive-tls (HIVE_TLS_CA_PATH) + hive-ci
(NODE_EXTRA_CA_CERTS) wiring derives correctly from it.
With self-signed TLS the gateway/forge serve a hive-CA-signed leaf and
forgejo's ROOT_URL is https://forge.<domain>. The CI runner's Node-based
actions (e.g. upload-artifact) POST to the ROOT_URL-derived artifact
endpoint and fail with "unable to verify the first certificate": Node uses
its own bundled CA bundle, not the system store, so it rejects the
self-signed chain. checkout etc. are fine — they hit the localhost http
registration URL.
Bind-mount the public hive CA cert (only ca.pem — never the hive-tls state
dir, which holds the CA + leaf private keys) into the hive-ci container and
set NODE_EXTRA_CA_CERTS on the runner service so every Node action trusts
it, hive-wide. Order container@hive-ci after the host hive-tls-ca service so
the cert exists before the bind-mount is set up. All gated on self-signed
mode; with an operator cert / ACME the public CA already validates and the
mount + env var are absent.
Eval-proven: self-signed → /run/hive-ca/ca.pem bind-mount (from
/var/lib/hive-tls/ca.pem), NODE_EXTRA_CA_CERTS=/run/hive-ca/ca.pem, and
container@hive-ci ordered after hive-tls-ca.service; certDir → all absent.
After a hive-ci restart the act_runner re-registers and immediately claims
any queued jobs, which can beat the in-container nix daemon coming up: its
nix-daemon.socket carries ConditionPathIsReadWrite=/nix/var/nix/daemon-socket
and is skipped until /nix/var is read-write, so the first nix-dependent
build dispatches into a cold daemon and hangs/retries (observed: a 55m48s
nix flake check vs the normal ~30s — a build-offload stall, not a code
failure; a non-nix step on the same runner passed in 1s, masking it).
Ordering the runner after/wants nix-daemon.socket does not help — a
condition-skipped unit satisfies systemd ordering immediately. Instead add
a blocking ExecStartPre that polls until the daemon actually answers
(nix store ping), placed with mkBefore so it runs ahead of the upstream
module's runner-registration ExecStartPre. The runner therefore cannot
register or claim jobs until nix is usable. Topology-agnostic (works
whether the daemon is in-container or a shared host socket); ~180s ceiling
then the unit fails cleanly rather than claiming jobs into a dead daemon.
Eval-proven: gitea-runner-hive's ExecStartPre is
[wait-nix-daemon, gitea-register-runner-hive] in that order.
Forgejo's Actions artifact API hands the runner an upload URL built from
forgejo's ROOT_URL — the public forge domain (forge.<domain>), not the
runner's 127.0.0.1:<httpPort> registration URL. The hive-ci container shares
host netns but has no resolver entry for the hive's own domains (the
gateway's localHostsEntry only touches the host), so actions/upload-artifact
fails with 'getaddrinfo ENOTFOUND forge.<domain>' while normal CI (checkout
via the localhost registration URL) is unaffected. Add a networking.extraHosts
entry mapping the forge domain to loopback so the upload reaches the local
forge via the gateway.
The runner hardcoded a 3h per-job timeout. Surface it as
services.hyperhive.forge.ci.jobTimeout so a stuck or runaway job is
bounded by a sane default while staying overridable for operators who
legitimately run longer jobs.
Default drops to 1h (comfortably covers a cold-cache nix build while
bounding a hang far sooner than the old 3h); raise it via the option
for genuinely long jobs. Accepts a Go duration string.
Resolves the stuck-runner concern from the CI-outage follow-up with a
runner-enforced job timeout rather than an external watchdog, per
operator direction.
Part of the tracker-tag cleanup: the hive convention is prose, not
issue-tracker tags, in code. Reword the 21 tags in the nix tree
(flake.nix + the hive-c0re/ci/gateway/network modules) to describe
the thing they pointed at, preserving the context without the tag.
Comment-only — no eval or logic change. Validated with nix fmt
(no reformatting) and nix flake check --no-build (all checks
evaluate clean); the full build check was skipped locally because
the shared remote builder is degraded, so CI will exercise the
build derivations once the runner recovers.
The runner's prefetch 401/403 diagnostic baked a literal issue tag into
its runtime journal output. Per the hive-wide rule (no issue/PR #NNN tags
in code), strip it — the message already explains the stale-core-token
condition and the operator remedy in full prose, so nothing is lost.
Two bugs that together kept the runner registration token from
refreshing (#1475):
1. Unit name: the prefetch's before/wantedBy/partOf targeted
nixos-container@hive-ci.service, but a declarative containers.<n> is
the host unit container@<n>.service (confirmed against the live
container@hive-matrix.service during the #1465 incident). The wrong
name made all three silent no-ops, so the partOf never bound — the
RemainAfterExit oneshot stayed 'active (exited)' and never re-ran on
nixos-container restart, leaving the stale token in place. Corrected
to container@hive-ci.service.
2. 401-hardening: the registration-token fetch used a bare curl -sf | jq,
so a forge-core-token that is stale/invalid for the current forge
(e.g. after a forge rebuild) 401s and fails silently every attempt for
the full 60s loop, then exits with a misleading 'core token absent or
forge unreachable'. Now capture the HTTP status and fail fast + loudly
on 401/403 with a clear message pointing at re-minting the core token.
The hive-ci-prefetch oneshot has RemainAfterExit=true and is wired to the
container only via wantedBy + before. Once it runs successfully it stays
'active (exited)' indefinitely, so systemd skips it on subsequent container
restarts. The runner-token file it wrote is never refreshed.
This breaks the runner after its first registration: the token written on
the first successful boot is either a placeholder (forge-core-token wasn't
ready yet) or a registration token that has since been consumed/rotated.
On the next container restart prefetch does not re-run, the stale token
persists, and the in-container register service fails with
'invalid_argument: runner registration token not found' — exactly the
symptom in the field (worked briefly, two orphan runners registered, then
permanently offline).
Add partOf = [ nixos-container@hive-ci.service ] so a container stop/restart
propagates to the prefetch unit, forcing it to re-run and fetch a fresh
registration token before the container comes back up. before= still orders
it ahead of the container start within the same transaction.
On first boot (or after a wipe) hive-c0re writes forge-core-token only
after the forge container starts and the admin is provisioned. This
lags hive-c0re.service becoming active. The previous code bailed
immediately with TOKEN=placeholder if the token file was absent,
causing the runner to fail registration with 'token not found'.
Fix: merge both waits (core-token file appearance + forge API ready)
into a single 60s retry loop. The early-bail path is removed; the
script only exits cleanly if .runner is valid (writes placeholder)
or a fresh registration token is obtained.
When .runner exists but core-token is absent after 60s, we keep the
existing credentials (safe — the runner holds valid creds; next boot
will validate properly).
Closes#1224 (which tracks #1221).
The gitea-actions-runner's host-scheme job processes use the
service's environment PATH, not the NixOS login-shell PATH.
Without this, 'nix flake check' and similar steps fail with
'nix: command not found'.
- add pkgs.nix to environment.systemPackages (ensures the binary
is in /run/current-system/sw/bin)
- set explicit PATH on gitea-runner-hive service covering both
/run/current-system/sw/bin and /nix/var/nix/profiles/default/bin
hive-ci-register.service now runs unconditionally on every boot (not
just when .runner is absent). Before fetching a registration token it
validates existing .runner credentials via the forge admin API:
- 200: runner still registered, write dummy token and exit
- 404: runner deleted from forge, purge .runner and re-register
- 000: forge unreachable, keep credentials (runner surfaces the error)
- other non-200 or malformed .runner: purge and re-register
Removes ConditionPathExists so stale credentials from a wiped forge
no longer block the runner indefinitely. Updates docs/ci.md to match.
The nixpkgs gitea-actions-runner module uses tokenFile as a systemd
EnvironmentFile, loaded before any ExecStartPre. Our previous
ExecStartPre override was too late — file didn't exist yet.
Fix: tmpfiles pre-creates /run/hive-ci/runner-token with a placeholder
on every boot. A new hive-ci-register.service oneshot (gated on
ConditionPathExists=!/var/lib/gitea-runner/hive/.runner) fetches the
real token and overwrites before gitea-runner-hive.service starts,
wired via After/Wants. On subsequent boots the placeholder is harmless
— nixpkgs register step exits early when .runner already exists.
Closes#918.
Per-instance package option does not exist in the nixpkgs module.
Package is configured at services.gitea-actions-runner.package.
Fixes build failure on main. Closes#916.
nix flake check pulls rust tools hermetically. curl/jq in preStart
use absolute store paths. nix is part of any NixOS system by default.
git is the only package genuinely needed at runtime.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removes the manual runnerTokenFile operator step. On first boot the
container fetches a runner registration token from Forgejo's admin
API using hive-c0re's existing admin token
(/var/lib/hyperhive/forge-core-token). A preStart script writes the
token to /run/hive-ci/runner-token; on subsequent boots it writes a
dummy (registered .runner creds take precedence anyway).
Operator bootstrap is now just `services.hyperhive.ci.enable = true`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds `services.hyperhive.ci` NixOS module that spins up a `hive-ci`
nixos-container running `gitea-actions-runner` against the hive-forge
Forgejo instance. Off by default; opt in with `ci.enable = true` after
generating a runner registration token in Forgejo.
Also adds `.forgejo/workflows/ci.yml` with four jobs: nix flake check,
formatting (nix fmt + cargo fmt), cargo test, and cargo clippy. Jobs
target the `hive-ci` runner label.
Container design mirrors hive-forge (shared host netns, non-ephemeral
state, loopback reach to forge). sandbox-fallback = true since nspawn
containers can't create user-namespaces for nix sandbox.
Closes#175.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
