Forgejo's Actions artifact API hands the runner an upload URL built from
forgejo's ROOT_URL — the public forge domain (forge.<domain>), not the
runner's 127.0.0.1:<httpPort> registration URL. The hive-ci container shares
host netns but has no resolver entry for the hive's own domains (the
gateway's localHostsEntry only touches the host), so actions/upload-artifact
fails with 'getaddrinfo ENOTFOUND forge.<domain>' while normal CI (checkout
via the localhost registration URL) is unaffected. Add a networking.extraHosts
entry mapping the forge domain to loopback so the upload reaches the local
forge via the gateway.
The runner hardcoded a 3h per-job timeout. Surface it as
services.hyperhive.forge.ci.jobTimeout so a stuck or runaway job is
bounded by a sane default while staying overridable for operators who
legitimately run longer jobs.
Default drops to 1h (comfortably covers a cold-cache nix build while
bounding a hang far sooner than the old 3h); raise it via the option
for genuinely long jobs. Accepts a Go duration string.
Resolves the stuck-runner concern from the CI-outage follow-up with a
runner-enforced job timeout rather than an external watchdog, per
operator direction.
Part of the tracker-tag cleanup: the hive convention is prose, not
issue-tracker tags, in code. Reword the 21 tags in the nix tree
(flake.nix + the hive-c0re/ci/gateway/network modules) to describe
the thing they pointed at, preserving the context without the tag.
Comment-only — no eval or logic change. Validated with nix fmt
(no reformatting) and nix flake check --no-build (all checks
evaluate clean); the full build check was skipped locally because
the shared remote builder is degraded, so CI will exercise the
build derivations once the runner recovers.
The runner's prefetch 401/403 diagnostic baked a literal issue tag into
its runtime journal output. Per the hive-wide rule (no issue/PR #NNN tags
in code), strip it — the message already explains the stale-core-token
condition and the operator remedy in full prose, so nothing is lost.
Two bugs that together kept the runner registration token from
refreshing (#1475):
1. Unit name: the prefetch's before/wantedBy/partOf targeted
nixos-container@hive-ci.service, but a declarative containers.<n> is
the host unit container@<n>.service (confirmed against the live
container@hive-matrix.service during the #1465 incident). The wrong
name made all three silent no-ops, so the partOf never bound — the
RemainAfterExit oneshot stayed 'active (exited)' and never re-ran on
nixos-container restart, leaving the stale token in place. Corrected
to container@hive-ci.service.
2. 401-hardening: the registration-token fetch used a bare curl -sf | jq,
so a forge-core-token that is stale/invalid for the current forge
(e.g. after a forge rebuild) 401s and fails silently every attempt for
the full 60s loop, then exits with a misleading 'core token absent or
forge unreachable'. Now capture the HTTP status and fail fast + loudly
on 401/403 with a clear message pointing at re-minting the core token.
The hive-ci-prefetch oneshot has RemainAfterExit=true and is wired to the
container only via wantedBy + before. Once it runs successfully it stays
'active (exited)' indefinitely, so systemd skips it on subsequent container
restarts. The runner-token file it wrote is never refreshed.
This breaks the runner after its first registration: the token written on
the first successful boot is either a placeholder (forge-core-token wasn't
ready yet) or a registration token that has since been consumed/rotated.
On the next container restart prefetch does not re-run, the stale token
persists, and the in-container register service fails with
'invalid_argument: runner registration token not found' — exactly the
symptom in the field (worked briefly, two orphan runners registered, then
permanently offline).
Add partOf = [ nixos-container@hive-ci.service ] so a container stop/restart
propagates to the prefetch unit, forcing it to re-run and fetch a fresh
registration token before the container comes back up. before= still orders
it ahead of the container start within the same transaction.
On first boot (or after a wipe) hive-c0re writes forge-core-token only
after the forge container starts and the admin is provisioned. This
lags hive-c0re.service becoming active. The previous code bailed
immediately with TOKEN=placeholder if the token file was absent,
causing the runner to fail registration with 'token not found'.
Fix: merge both waits (core-token file appearance + forge API ready)
into a single 60s retry loop. The early-bail path is removed; the
script only exits cleanly if .runner is valid (writes placeholder)
or a fresh registration token is obtained.
When .runner exists but core-token is absent after 60s, we keep the
existing credentials (safe — the runner holds valid creds; next boot
will validate properly).
Closes#1224 (which tracks #1221).
The gitea-actions-runner's host-scheme job processes use the
service's environment PATH, not the NixOS login-shell PATH.
Without this, 'nix flake check' and similar steps fail with
'nix: command not found'.
- add pkgs.nix to environment.systemPackages (ensures the binary
is in /run/current-system/sw/bin)
- set explicit PATH on gitea-runner-hive service covering both
/run/current-system/sw/bin and /nix/var/nix/profiles/default/bin
hive-ci-register.service now runs unconditionally on every boot (not
just when .runner is absent). Before fetching a registration token it
validates existing .runner credentials via the forge admin API:
- 200: runner still registered, write dummy token and exit
- 404: runner deleted from forge, purge .runner and re-register
- 000: forge unreachable, keep credentials (runner surfaces the error)
- other non-200 or malformed .runner: purge and re-register
Removes ConditionPathExists so stale credentials from a wiped forge
no longer block the runner indefinitely. Updates docs/ci.md to match.
The nixpkgs gitea-actions-runner module uses tokenFile as a systemd
EnvironmentFile, loaded before any ExecStartPre. Our previous
ExecStartPre override was too late — file didn't exist yet.
Fix: tmpfiles pre-creates /run/hive-ci/runner-token with a placeholder
on every boot. A new hive-ci-register.service oneshot (gated on
ConditionPathExists=!/var/lib/gitea-runner/hive/.runner) fetches the
real token and overwrites before gitea-runner-hive.service starts,
wired via After/Wants. On subsequent boots the placeholder is harmless
— nixpkgs register step exits early when .runner already exists.
Closes#918.
Per-instance package option does not exist in the nixpkgs module.
Package is configured at services.gitea-actions-runner.package.
Fixes build failure on main. Closes#916.
nix flake check pulls rust tools hermetically. curl/jq in preStart
use absolute store paths. nix is part of any NixOS system by default.
git is the only package genuinely needed at runtime.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removes the manual runnerTokenFile operator step. On first boot the
container fetches a runner registration token from Forgejo's admin
API using hive-c0re's existing admin token
(/var/lib/hyperhive/forge-core-token). A preStart script writes the
token to /run/hive-ci/runner-token; on subsequent boots it writes a
dummy (registered .runner creds take precedence anyway).
Operator bootstrap is now just `services.hyperhive.ci.enable = true`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds `services.hyperhive.ci` NixOS module that spins up a `hive-ci`
nixos-container running `gitea-actions-runner` against the hive-forge
Forgejo instance. Off by default; opt in with `ci.enable = true` after
generating a runner registration token in Forgejo.
Also adds `.forgejo/workflows/ci.yml` with four jobs: nix flake check,
formatting (nix fmt + cargo fmt), cargo test, and cargo clippy. Jobs
target the `hive-ci` runner label.
Container design mirrors hive-forge (shared host netns, non-ephemeral
state, loopback reach to forge). sandbox-fallback = true since nspawn
containers can't create user-namespaces for nix sandbox.
Closes#175.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
