docs/4-implementation/scheduler/ci.md described both the swarm-ci-runner product (hosting, registration, security, store maintenance) and this repo's own pipeline (which jobs run on a PR, the local equivalents, the nightly VM tests). The second part is project process: it moves, whole and in order, to docs/5-project/process/ci-checks.md, with a pointer left behind, an entry in the docs index and the one doc comment in hive-forge that named the old section updated.
323 lines
19 KiB
Markdown
323 lines
19 KiB
Markdown
# swarm-ci-runner: Forgejo Actions Runner
|
|
|
|
The `swarm-ci-runner` module runs the swarm's Forgejo Actions runner in a
|
|
`swarm-ci-runner` nixos-container, executing CI jobs from
|
|
`.forgejo/workflows/ci.yml` on every PR. It's a swarm service: it runs on
|
|
whichever host sets it, and needs neither the forge nor a hive on that host.
|
|
|
|
## For operators
|
|
|
|
**Enabling it's one line**: `services.hyperhive.deploy.ci-runner.enable = true`
|
|
in the NixOS config of the host that should run it. No manual token
|
|
provisioning: swarm-controller publishes the forge's runner registration token
|
|
to the secret store, and the runner reads it with a store identity of its own.
|
|
On the store's host `glue-bao-tls.nix` mints that identity; on any other host, copy
|
|
`ci-runner.pem` and `ci-runner-key.pem` from the store host's
|
|
`/var/lib/swarm-bao-pki/` and name them in
|
|
`services.hyperhive.deploy.bao.ciRunnerClientCertFile` and
|
|
`ciRunnerClientKeyFile`.
|
|
|
|
Two things worth knowing before you rely on it:
|
|
|
|
- **Only run CI for contributors you trust.** Builds run **unsandboxed**
|
|
(an nspawn limitation, not a choice) — a malicious build script in a
|
|
PR could make network requests or write to the container filesystem
|
|
during the build. Fine for a small trusted-team hive where everyone
|
|
already has forge access; if you take outside/fork contributions,
|
|
gate CI behind Forgejo's fork-PR-approval setting or restrict the
|
|
workflow to push-only triggers (forks can't push to your branches).
|
|
See _Security_ below for the full threat model.
|
|
- **Watch your disk.** CI builds through the host's nix store with no
|
|
automatic garbage collection of its own — a busy CI day can fill
|
|
`/nix/store` until every job fails with `ENOSPC`. Add the daily +
|
|
disk-pressure GC config from _Host store maintenance_ below to your
|
|
host's NixOS config (not optional if you plan to actually use this).
|
|
|
|
Everything below this point is implementation detail: registration
|
|
internals, container design, and the full security writeup.
|
|
|
|
## This repo's own checks
|
|
|
|
Which jobs run on this repo's pull requests, how to run them locally and how the
|
|
nightly VM tests work are project process, in
|
|
[the repo's CI checks](../../5-project/process/ci-checks.md). This page covers the
|
|
runner itself.
|
|
|
|
## Configuration reference
|
|
|
|
`services.hyperhive.deploy.ci-runner.enable = true` on the host that runs it
|
|
(see _For operators_ above). The runner reaches the forge at
|
|
`https://<swarm.forge.domain>`, so that name has to resolve from the host: the
|
|
gateway's dnsmasq answers it locally where the forge runs, and forwards it to
|
|
the operator's DNS everywhere else.
|
|
|
|
Optional tuning under `services.hyperhive.deploy.ci-runner`: `name` (runner
|
|
name in the forge admin panel), `concurrency` (parallel job capacity), `labels`
|
|
(workflow targeting; the default `hive-ci:host` is what this repo's workflows
|
|
name), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration string
|
|
for example `"3h"` — a job that exceeds it's killed so a hung or runaway build
|
|
can't hold the runner's single slot indefinitely).
|
|
|
|
The old `services.hyperhive.deploy.forgejo.ci.{enable,name,concurrency,labels,package,jobTimeout}`
|
|
and `services.hyperhive.swarm.forge.ci.{enable,name,concurrency,labels,package}`
|
|
are still accepted: each forwards to the same-named option under
|
|
`deploy.ci-runner`, and evaluation warns naming both paths. Set the new name to
|
|
silence the warning. The two `name` defaults differ (`hive-ci` before,
|
|
`swarm-ci-runner` now), so a host that never set the old `name` shows up under
|
|
the new default.
|
|
|
|
The forwarded `enable` carries the new module's requirements, not the old
|
|
module's: evaluation refuses a host that sets only the old `enable` unless
|
|
`services.hyperhive.deploy.bao.ciRunnerClientCertFile` and
|
|
`ciRunnerClientKeyFile` are both set. They default to the identity
|
|
`glue-bao-tls.nix` mints when `services.hyperhive.deploy.bao.enable` is true;
|
|
elsewhere place them as described under _For operators_. The runner no longer
|
|
needs `deploy.forgejo.enable` or a hive on its host.
|
|
|
|
## Container design
|
|
|
|
- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI.
|
|
- **Non-ephemeral**: runner credentials (`.runner`) persist across restarts, written to the container's state dir on registration and reused after.
|
|
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/5-project/process/gotchas.md`.
|
|
- **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path. Only `swarm-ci-runner-register.service` loads the registration token; the daemon unit and the jobs it runs have it neither in their environment nor among their credentials.
|
|
|
|
## Registration
|
|
|
|
1. **swarm-controller** (`swarm-controller/src/forge/runner_token.rs`) reads
|
|
the forge's instance-wide token from
|
|
`GET /api/v1/admin/runners/registration-token` at start and every five
|
|
minutes, and writes it to the secret store at
|
|
`swarm/services/ci-runner/registration-token` when it differs from the
|
|
value it last wrote. Its store grant on that path is write-only.
|
|
2. **`swarm-bao-ci-runner.service`** on the runner's host logs in to the store
|
|
with the `ci-runner` leaf, whose `swarm-ci-runner` role reads that one path
|
|
and nothing else, retrying every 30 seconds until the token is there. It
|
|
writes the token to a root-only file in `/run/swarm-ci-runner` on the host,
|
|
which the container sees read-only, and restarts the runner when the value
|
|
changes. `/run` is tmpfs, so no copy reaches persistent disk; after a reboot
|
|
the unit fetches the token again, and until the store answers only a
|
|
runner that has no `.runner` yet waits. It runs at boot, on deploy, and every
|
|
ten minutes on a timer.
|
|
3. **`swarm-ci-runner-register.service`** in the container is the only unit
|
|
that loads that file, with `LoadCredential=`. systemd hands a unit's
|
|
credentials to all of its processes, and for a `host` label act_runner
|
|
runs every job step as a process of the daemon with the daemon's whole
|
|
environment, so the daemon unit carries the token in neither. The
|
|
register unit is a oneshot under the daemon's dynamic user and state
|
|
directory. It reads the token from `$CREDENTIALS_DIRECTORY` and runs
|
|
`act_runner register` only when `.runner` is missing, so the `.runner` it
|
|
writes is the daemon's own. **The runner unit** (`gitea-runner-swarm`)
|
|
starts it in a pre-start step and runs the daemon once it succeeds.
|
|
A root step of the register unit, ahead of registering, removes `.runner`
|
|
in three cases:
|
|
- The forge has no record of the runner. Forgejo answers every call from an
|
|
unknown runner with `unregistered runner`, and the daemon exits 1 when its
|
|
startup `Declare` gets that answer. On the next start the step finds the
|
|
message in the daemon's journal since its last start, and registers again.
|
|
A daemon that's already running when an admin deletes its runner keeps polling
|
|
and logging the error without exiting; the next restart of the unit or
|
|
container registers it again.
|
|
- `.runner` names another forge address than the configured one.
|
|
- `.runner` names other labels than the configured ones, compared in the
|
|
form the daemon writes back: `<name>:<scheme>`, with `host` for a
|
|
label that names no scheme.
|
|
|
|
A changed token alone doesn't register again: the daemon authenticates with
|
|
the per-runner token in `.runner`. Without `.runner` and with no token yet
|
|
the register unit fails and the runner unit's pre-start step with it;
|
|
`Restart=on-failure` on the runner unit starts both again every 15 seconds,
|
|
with no start limit.
|
|
|
|
Host jobs run as the runner's own user, so they can still read `.runner`
|
|
and act as this runner; see the security section below.
|
|
|
|
A new registration is a new runner on the forge: the previous entry stays in
|
|
the forge admin panel as offline until an admin deletes it. The runner the old
|
|
`hive-ci` module registered is one such entry, since this runner registers
|
|
under its own name and token. Delete it under Site administration, Actions,
|
|
Runners.
|
|
|
|
## Actions mirrors
|
|
|
|
<!-- vale write-good.Passive = NO -->
|
|
|
|
With `services.hyperhive.deploy.forgejo.actionsFromSelf` enabled, the swarm
|
|
controller seeds `actions/checkout` and `actions/upload-artifact` pull-mirrors
|
|
(from `github.com/actions/*`) on the forge and Forgejo's `DEFAULT_ACTIONS_URL`
|
|
points at the forge itself. CI `uses: actions/<name>@vN` steps then resolve on
|
|
the forge, with no external DNS on the CI critical path. A workflow that names
|
|
any other bare `actions/*` action fails at job setup with "repository not
|
|
found" until that action joins the list in `nix/host-modules/hive-forge/default.nix`.
|
|
|
|
A forge host that runs the swarm controller therefore resolves those actions
|
|
from itself, with or without a runner on that host.
|
|
Set `actionsFromSelf = false` there to resolve actions from `data.forgejo.org`
|
|
and seed no mirror.
|
|
|
|
The option defaults to whether the forge's host also runs the swarm controller
|
|
(`services.hyperhive.deploy.swarm-controller.enable`; set by `glue-forge-controller.nix`): on when it does, off
|
|
otherwise. Setting it `true` on a forge host without the controller leaves
|
|
the actions unresolved, because nothing seeds the mirrors; evaluation
|
|
warns about it.
|
|
|
|
<!-- vale write-good.Passive = YES -->
|
|
|
|
The **swarm controller** seeds the mirror in its swarm-wide forge-objects
|
|
pass (`swarm-controller/src/forge/objects.rs`), beside the orgs it ensures.
|
|
`glue-forge-controller.nix` forwards the effective mirror list as
|
|
`SWARM_CONTROLLER_FORGE_MIRRORS` in the `swarm-controller` service
|
|
environment (JSON-encoded `[{upstream, dest}]` list). It reads the list of
|
|
the host the controller runs on: mirrors declared on any other host aren't
|
|
seeded, and evaluation warns about them.
|
|
|
|
**The hyperhive mirror**: the same list carries `hyperhive/hyperhive`, from
|
|
`services.hyperhive.deploy.forgejo.hyperhiveUpstream` (default
|
|
`https://forge.darkest.space/hyperhive/hyperhive` on a host that runs the
|
|
controller, unset elsewhere). The controller creates the mirror only when
|
|
`hyperhive/hyperhive` doesn't exist, and it leaves a repo that exists but is
|
|
not a mirror unchanged. `deploy.forgejo.mirrors` takes precedence for the same
|
|
`dest`.
|
|
|
|
**General-purpose mirrors**: you can pre-seed any external repo as a
|
|
pull-mirror via `services.hyperhive.deploy.forgejo.mirrors`:
|
|
|
|
```nix
|
|
services.hyperhive.deploy.forgejo.mirrors = [
|
|
{ upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; }
|
|
{ upstream = "https://github.com/example/tool"; dest = "mirrors/tool"; }
|
|
];
|
|
```
|
|
|
|
The controller creates each entry as a real Forgejo pull-mirror — not a one-off
|
|
clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack`
|
|
request), so a DNS blip during that sync will propagate back to the
|
|
runner as a hard `git clone` failure. The controller
|
|
autocreates the `<owner>` org in `dest`. Keep mirror dests out of the
|
|
hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`)
|
|
to avoid provisioning collisions.
|
|
|
|
## Security: unsandboxed builds and trusted contributors
|
|
|
|
**The runner should only run CI for trusted contributors.** The security boundary is weaker than it looks:
|
|
|
|
### What unsandboxed builds mean
|
|
|
|
nspawn containers can't create user-namespaces, so the module sets `nix.settings.sandbox-fallback = true` in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.
|
|
|
|
A malicious `default.nix` or build script in a PR can therefore:
|
|
|
|
- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at `https://<forge.domain>` (no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic).
|
|
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
|
|
|
|
No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which only `swarm-ci-runner-register.service` loads, so a job step can't read it. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls.
|
|
|
|
Note: `nix flake check --no-build` (eval-only) reduces the exposure but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The CI workflow's `nix flake check` job builds every check derivation, which is the higher-risk path.
|
|
|
|
### Mitigation
|
|
|
|
For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway.
|
|
|
|
For repos with external contributors or fork PRs:
|
|
|
|
- Use Forgejo's **fork PR approval workflow** (`repository.settings` → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
|
|
- Or restrict the CI workflow trigger to push events on branches (not `pull_request` from forks) — forks can't push to upstream branches.
|
|
|
|
The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.
|
|
|
|
## Host store maintenance (recommended)
|
|
|
|
The CI runner builds derivations through the **host** nix-daemon — the
|
|
`swarm-ci-runner` container shares the host store and has no daemon of its own. Build
|
|
outputs accumulate in `/nix/store` with no automatic collection, and a busy
|
|
CI day can fill the disk until every job fails fast with `ENOSPC`.
|
|
|
|
Store GC is a **host-level** concern, so it belongs in the host's own NixOS
|
|
configuration, not in the hyperhive service modules — a single service should
|
|
not reach out and change the host's global nix-daemon options. The module
|
|
only warns at `nixos-rebuild` time when the host sets neither
|
|
`nix.gc.automatic` nor `nix.settings.min-free`. Add the following to your
|
|
host config:
|
|
|
|
```nix
|
|
{
|
|
# Daily GC: delete store paths not referenced by a live root and older
|
|
# than a day. Keeps the store bounded between builds.
|
|
#
|
|
# `randomizedDelaySec` because a fixed daily time lands on the same wall
|
|
# clock as whatever else runs nightly. A collection that overlaps a CI
|
|
# job doesn't fail it — the runner builds through the host nix-daemon,
|
|
# so an in-flight build's temp roots are protected — but the first build
|
|
# after a collection rebuilds what the collection deleted, so a run that
|
|
# lands just behind one pays for it in wall time.
|
|
nix.gc = {
|
|
automatic = true;
|
|
dates = "daily";
|
|
randomizedDelaySec = "45min";
|
|
options = "--delete-older-than 1d";
|
|
};
|
|
|
|
# Disk-pressure GC: when free space drops below min-free mid-build, the
|
|
# daemon collects garbage up to max-free before continuing. This is the
|
|
# real-time net the daily timer can't provide — a same-day build burst is
|
|
# what fills the disk. Tune to your disk size.
|
|
nix.settings.min-free = 20 * 1024 * 1024 * 1024; # 20 GiB
|
|
nix.settings.max-free = 50 * 1024 * 1024 * 1024; # 50 GiB
|
|
}
|
|
```
|
|
|
|
**Measure a collection on your own hosts before relying on that stagger.**
|
|
`randomizedDelaySec` spreads the _start_, not the run, and the sweep over
|
|
unused `.links` entries dominates the run — which scales with the size of the
|
|
store, not with what this collection deletes. On a large store it can take most
|
|
of an hour, so two hosts both set to `daily` may overlap however much jitter you
|
|
give them. `journalctl -u nix-gc.service` reports each run's wall clock.
|
|
|
|
**Remote builders:** if CI dispatches builds to a remote builder (for example via
|
|
`nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_
|
|
store, so apply the same GC config wherever the builder runs —
|
|
GC on the coordinator host won't reclaim space on the builder.
|
|
|
|
⚠️ Two stores also means the temp-root protection above **doesn't span
|
|
them**. Each daemon holds roots only for the builds it runs, so a
|
|
collection on the _builder_ knows nothing about a coordinator build
|
|
waiting on a path there. Stagger the two hosts' GC windows rather than
|
|
giving them the same `dates`, and prefer `min-free`/`max-free` on the
|
|
builder — disk-pressure collection at least runs while nothing is mid-copy.
|
|
|
|
## References
|
|
|
|
- `nix/host-modules/swarm-ci-runner.nix`: runner configuration, token fetch, re-registration step, container setup.
|
|
- `swarm-controller/src/forge/runner_token.rs`: publishes the registration token.
|
|
- `.forgejo/workflows/ci.yml`: workflow definition.
|
|
- `docs/5-project/process/gotchas.md`: nix sandboxing limitations in containers.
|
|
|
|
## Host setup
|
|
|
|
One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that
|
|
sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a
|
|
runner executes pull-request code, so running one is a decision of its own.
|
|
|
|
It needs neither the forge nor a hive on its host. It reaches the forge at
|
|
`https://<swarm.forge.domain>`, and registers with the token swarm-controller
|
|
publishes to the secret store, which it reads under its own `ci-runner` leaf.
|
|
On a host without the store, that leaf is the one file to copy.
|
|
|
|
The runner's built-in Prometheus endpoint is on, bound to `127.0.0.1:9101` in
|
|
the container's own network namespace. The container's journal forwarder
|
|
scrapes it (`services.hyperhive.swarmJournalForwarder.extraScrapeTargets`) and
|
|
pushes the series to the swarm collector under the job `gitea-runner`, so
|
|
nothing listens on the bridge. Series, all prefixed `gitea_runner_`:
|
|
`job_running`, `capacity`, `job_capacity_utilization_ratio`, `uptime_seconds`,
|
|
`info` (labels `version` and `name`), `job_total{status}`,
|
|
`job_duration_seconds`, `poll_fetch_total{result}`,
|
|
`poll_fetch_duration_seconds`, `poll_backoff_seconds`, `report_log_*`,
|
|
`report_state_*` and `client_errors_total`. A pull-request job runs in the same
|
|
namespace and can read the endpoint, which carries counts, durations, the
|
|
runner's name and its version, and no secrets.
|
|
|
|
The **ci runner** board in Grafana shows these series. Its **Runner** filter
|
|
(the series' `service.instance.id`, the scrape address) narrows every panel,
|
|
and uptime, job slots and slot use show one series per runner, so a runner
|
|
registered by hand appears beside the swarm's once its metrics reach the
|
|
store under its own instance id. The forge's **CI runs** tab is the other
|
|
view: run times and queue wait of every run, whichever runner ran it.
|