The token file is 0600 root-owned and the precondition ran as the unit's
dynamic gitea-runner user, so it could not read it -- and it did not fail
closed. `[ -s ]` succeeds on a stat alone, grep then died with EACCES,
and `! grep` turned that error into true, so the gate exited 0 on a file
it had never read. It has reported 'a real token is present' without
looking for its entire existence, which is why the runner kept retrying
instead of holding still and saying why.
Prefix the step with + so it runs with full privileges. This grants the
runner nothing new: tokenFile becomes EnvironmentFile=, which systemd
already reads as root before dropping privileges, so the payload never
passes through an unprivileged reader either way -- the + only lets the
gate observe what systemd observes.
Also assert readability separately and loudly, rather than folding it
into the placeholder test: an unreadable file and a real token produced
the same verdict, so 'I could not look' was indistinguishable from 'I
looked and it is fine'.
NODE_EXTRA_CA_CERTS beside it is Node's and additive. gitea-runner is Go,
whose trust store is replacing: it reads SSL_CERT_FILE and trusts only
what that names. So the container trusted the hive CA for its Node
actions and never for the daemon itself.
The daemon then fails startup with x509 unknown authority the moment it
reaches a TLS endpoint, crash-loops on Restart=on-failure, and no job is
picked up -- every pull request just sits at Waiting to run.
Not claimed as the whole cause of that outage: which address the daemon
holds is a separate question. The trust gap is real under every
explanation, and adding a CA to one unit cannot break a working path.
Forgejo is a swarm-global service, so its operator-facing host options
move to services.hyperhive.swarm.forge (and .swarm.forge.ci) as the
first of the namespace consolidation.
Existing hive configs keep evaluating: swarm-renames.nix maps every
moved leaf with mkRenamedOptionModule, which also emits a deprecation
warning naming both the old and new path, so an operator is told what to
rename rather than discovering it from a failed eval.
The per-agent hyperhive.forge.url does NOT move. It is a client pointer
at whatever forge an agent talks to - it shares a word with the service
and nothing else, and the two are already documented as separate option
surfaces.
Verified by evaluating the host module, since no Rust gate evaluates
nix: setting the old paths and reading the new ones yields the values
(httpPort 3999, ci.concurrency 7), and config.warnings carries the
rename notice.
Container nix invocations hard-failed whenever the remote builder
muede-pc2 was unreachable, while the identical build on the host
succeeded. Both go through the same host nix-daemon, so the difference
looked impossible.
The cause is that `fallback` is a client-side option: the nix client
transmits it to the daemon per connection (`tryFallback` in setOptions),
so the caller decides whether a failed remote dispatch may degrade to a
local build — even when the build itself runs on the host daemon under
NIX_REMOTE=daemon. Only genuinely daemon-side settings such as
`builders` are inherited from the host. The containers never set
`fallback`, so they took nix's default of false.
Set it in the agent-container base module and in the CI container, and
correct the hive-ci comment that claimed fallback was inherited from
the host daemon along with buildMachines and max-jobs.
Verified in an agent container: `nix fmt` fails outright on the remote
store's connection reset, while the same command with fallback enabled
reports the same connection error and then builds locally and succeeds.
CI's nix flake check ran in hive-ci's OWN in-container nix-daemon, which
offloads to the pc2 remote builder and HARD-FAILS when pc2 is
unreachable (Connection-reset) — reddening every PR's queue hive-wide.
The host daemon builds fine in the same situation (buildMachines +
max-jobs>=1 + fallback → local build when pc2 is down), and the agent
containers already route through it.
Give hive-ci the same wiring: bind-mount the host nix-daemon socket dir
into the container, set NIX_REMOTE=daemon, and disable the container's
own nix-daemon service + socket. Now CI builds through the host daemon —
pc2 offload when it's up, graceful local fallback when it's down. Drops
the now-moot in-container wait-nix-daemon precond. Needs an operator
rebuild to apply.
Completes #2415. Registration no longer gates container@hive-ci start:
- Retire the host-side hive-ci-prefetch.service (+ its 100-line script and
the now-dead coreTokenPath/containerRoot let-bindings) — the forge round-trip
it did on the boot path now lives in hive-c0re (forge/ci_runner.rs), run out
of band during the startup sweep.
- Drop the container@hive-ci TimeoutStartSec = mkForce 180s band-aid that
papered over that boot-path wait tripping the nspawn start timeout (the ~60s
restart loop of #2410).
- gitea-runner-hive gains an ExecStartPre precond (ahead of the nix-daemon
wait) that fails fast unless it is already registered (.runner present) or a
real, non-placeholder token is in place — so missing creds just hold the
runner down instead of blocking the container. Restart=on-failure (no start
limit) self-heals it once hive-c0re writes the token; c0re's explicit restart
is the primary path.
- Set HYPERHIVE_FORGE_CI_ENABLED=1 on hive-c0re.service so the sweep registers.
The tmpfiles TOKEN=placeholder seed + read-only bind-mount are unchanged; the
core admin token still never enters the container.
hive-ci and hive-forge both bind the runtime-generated hive CA cert
read-only and order their container@ unit after hive-tls-ca.service so
the bind source exists before nspawn sets the mount up — the same
bind-mount + ordering + rationale duplicated verbatim in two modules.
Extract that language-agnostic half into a pure helper,
nix/host-modules/lib/hive-ca-trust.nix, taking a container name and
returning { useSelfSigned, caContainerPath, bindMount, containerOrdering }.
The per-runtime consumption stays at each call site (hive-ci's additive
NODE_EXTRA_CA_CERTS, hive-forge's Go SSL_CERT_FILE concat). hive-ci folds
containerOrdering into its existing mkMerge alongside the TimeoutStartSec
bump.
The helper is a pure function, not a module: host-modules/default.nix is
an explicit aggregator (not a glob) and the docs eval imports that same
aggregator, so the lib/ file is never picked up as a module. A third
outbound-TLS-trusting container no longer means a third copy-paste.
gitea-runner registration (hive-ci-prefetch, host-side) sits on the
boot-critical path -- nspawn readiness is gated on the runner
registering, itself a forge + core-token round trip that can wait up
to 60s for the core token. The default ~60s TimeoutStartSec can trip
mid-register (especially right after a .runner purge, since every
boot re-registers from scratch), killing the half-started container
and triggering a restart loop until the token/forge settle. Bump to
180s so one register attempt has room to finish.
