814edd1d67
Each service section lost its container, credential and metrics detail to a Host setup section on its own implementation page, unchanged apart from links. The four agentAuth option rows that sat above the page title moved into the telemetry page's metrics option table.
422 lines
25 KiB
Markdown
422 lines
25 KiB
Markdown
# swarm-ci-runner: Forgejo Actions Runner
|
|
|
|
The `swarm-ci-runner` module runs the swarm's Forgejo Actions runner in a
|
|
`swarm-ci-runner` nixos-container, executing CI jobs from
|
|
`.forgejo/workflows/ci.yml` on every PR. It's a swarm service: it runs on
|
|
whichever host sets it, and needs neither the forge nor a hive on that host.
|
|
|
|
## For operators
|
|
|
|
**Enabling it's one line**: `services.hyperhive.deploy.ci-runner.enable = true`
|
|
in the NixOS config of the host that should run it. No manual token
|
|
provisioning: swarm-controller publishes the forge's runner registration token
|
|
to the secret store, and the runner reads it with a store identity of its own.
|
|
On the store's host `glue-bao-tls.nix` mints that identity; on any other host, copy
|
|
`ci-runner.pem` and `ci-runner-key.pem` from the store host's
|
|
`/var/lib/swarm-bao-pki/` and name them in
|
|
`services.hyperhive.deploy.bao.ciRunnerClientCertFile` and
|
|
`ciRunnerClientKeyFile`.
|
|
|
|
Two things worth knowing before you rely on it:
|
|
|
|
- **Only run CI for contributors you trust.** Builds run **unsandboxed**
|
|
(an nspawn limitation, not a choice) — a malicious build script in a
|
|
PR could make network requests or write to the container filesystem
|
|
during the build. Fine for a small trusted-team hive where everyone
|
|
already has forge access; if you take outside/fork contributions,
|
|
gate CI behind Forgejo's fork-PR-approval setting or restrict the
|
|
workflow to push-only triggers (forks can't push to your branches).
|
|
See _Security_ below for the full threat model.
|
|
- **Watch your disk.** CI builds through the host's nix store with no
|
|
automatic garbage collection of its own — a busy CI day can fill
|
|
`/nix/store` until every job fails with `ENOSPC`. Add the daily +
|
|
disk-pressure GC config from _Host store maintenance_ below to your
|
|
host's NixOS config (not optional if you plan to actually use this).
|
|
|
|
Everything below this point is implementation detail: registration
|
|
internals, container design, and the full security writeup.
|
|
|
|
## CI checks
|
|
|
|
<!-- vale write-good.Passive = NO -->
|
|
|
|
The jobs run on every PR (and on `workflow_dispatch` for manual
|
|
re-triggers), defined in
|
|
[`.forgejo/workflows/ci.yml`](../../../.forgejo/workflows/ci.yml). The first four
|
|
below are required checks (forge branch protection) — a hit on any of them
|
|
blocks merge. The two vale jobs split what used to be a single check: the
|
|
error-only one exists so it _can_ become a required check without also
|
|
gating on the full job's standing warning/suggestion backlog, but flipping
|
|
that switch on the forge is a separate operator action, not something this
|
|
table can assert ahead of it happening.
|
|
|
|
<!-- vale write-good.Passive = YES -->
|
|
|
|
| Job | What it runs | Required? |
|
|
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------- |
|
|
| **nix flake check** | `scripts/ci-flake-check.sh`: nix-eval-jobs, then builds each check (treefmt, clippy, tests, module eval); no VM tests | yes |
|
|
| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) | yes |
|
|
| **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) | yes |
|
|
| **doc-pointer lint** | flags `docs/…md` paths and relative markdown links that no longer resolve (`scripts/check-doc-refs.sh`) | yes |
|
|
| **attribution-trailer lint** | flags `Co-Authored-By` trailers with Claude/Anthropic attribution in PR commits (`scripts/check-attribution-trailers.sh`) | not yet — intended to be |
|
|
| **prose lint (vale, errors)** | vale, `docs/` tree, error-severity rule hits only | not yet — intended to be |
|
|
| **prose lint (vale)** | vale, `docs/` tree, full severity range (warnings + suggestions too) | no — advisory, standing backlog |
|
|
|
|
`hive-forge ci-rerun --pr N` dispatches a `workflow_dispatch` retrigger
|
|
without an empty commit.
|
|
|
|
<!-- vale write-good.Passive = NO -->
|
|
|
|
**`ci-rerun --pr` verifies the code but doesn't reliably move the PR's own
|
|
status checks.** Measured directly (raw `GET /repos/.../actions/tasks`
|
|
JSON): a genuine PR-triggered run carries `event: pull_request` with the
|
|
`#<n>` pseudo-ref as its `head_branch`; a `ci-rerun --pr`-dispatched run
|
|
carries `event: workflow_dispatch` with the real branch name instead.
|
|
Forgejo's PR commit-status tracking appears scoped to `pull_request`-event
|
|
runs specifically, so a dispatched run — even a genuinely successful one —
|
|
never writes to that status slot. Reproduced with three dispatches on one
|
|
PR over 20+ minutes, all genuinely green, the PR's own status unmoved
|
|
throughout. **Always re-check `pr status` after a dispatch and believe
|
|
what it says — don't push a commit just to unwedge it, that's the
|
|
empty-commit anti-pattern this verb exists to avoid.** If `hive-forge
|
|
ci-log --run <n> --job 0` shows the code green (verdict is the
|
|
last line) but the status is stuck, ask the operator to select "rerun" in
|
|
the forge web UI — it's CSRF-gated, so only they can do it.
|
|
|
|
<!-- vale write-good.Passive = YES -->
|
|
|
|
### Running checks locally
|
|
|
|
Don't run `nix flake check` directly — it dispatches to the shared build farm and
|
|
wastes a remote-builder slot. Use the devshell equivalents instead:
|
|
|
|
```sh
|
|
nix develop -c cargo clippy --all-targets -- -D warnings
|
|
nix develop -c cargo test
|
|
nix fmt # authoritative; the formatter set is nix/treefmt.nix
|
|
sh scripts/check-issue-refs.sh # tracker-tag lint
|
|
sh scripts/check-comment-blocks.sh # comment-block lint
|
|
```
|
|
|
|
`scripts/pre-push` provides a git pre-push hook that automates the two
|
|
lint checks. Install it once per clone:
|
|
|
|
```sh
|
|
ln -sf ../../scripts/pre-push .git/hooks/pre-push
|
|
```
|
|
|
|
After that, any `git push` automatically runs both lints and aborts with a
|
|
diagnostic if either fails — catching the issue locally before CI sees it.
|
|
Note that the hook does **not** run `cargo clippy` or `cargo test` (those are
|
|
slow); run those manually before pushing Rust changes.
|
|
|
|
### VM tests
|
|
|
|
Tests that boot NixOS VMs are the `vmTests.<system>.<name>` flake output
|
|
(`nix/vm-tests.nix`), not `checks`, so the per-PR **nix flake check** job never
|
|
builds them. It does build the `vm-test-typecheck-<name>` checks
|
|
(`nix/vm-test-typecheck.nix`): the type check and lint the test driver runs over
|
|
a test script, on the script text alone. They evaluate no node configuration
|
|
and build no VM. They cover all seven tests. `semi-spread` and `swarm-builder-offline`
|
|
read node config or an agent toplevel into their script, so each builds the script
|
|
with stand-in values (`passthru.typecheckScript`), from the same function as the real
|
|
one.
|
|
`vm-test-typecheck-control` passes only when the checker rejects a known type error, which shows the
|
|
checker resolves `test_driver`. [`.forgejo/workflows/vm-nightly.yml`](../../../.forgejo/workflows/vm-nightly.yml)
|
|
builds all of them nightly at 02:30; a failing test fails that run. To run one
|
|
by hand, start the workflow from the forge UI (Actions, vm-nightly, Run
|
|
workflow, optional `test` input), or run
|
|
`nix build .#vmTests.x86_64-linux.<name>` on a host with KVM. Current tests:
|
|
`agent-ephemeral-rootfs` (the switch of an agent container to an ephemeral
|
|
root, through the real hive-priv unit), `hive-squid`, and `wg-hive` (two hives
|
|
peered over the `wg-hive` mesh: handshake, traffic, and a wrong-key peer that
|
|
must not handshake).
|
|
|
|
The `full-setup` script is plain Python in `nix/vm-tests/full-setup.py`, with no
|
|
Nix interpolation. Commands the script runs in the VM find their binaries on the
|
|
node's `environment.systemPackages`. Build-time values (socket path, forge
|
|
domain, store paths) arrive as `FULL_SETUP_*` environment variables, set in
|
|
`nix/vm-tests/full-setup.nix` through `overrideTestDerivation`. To add a value,
|
|
set one there and read it with `os.environ` in the script. `nix fmt` formats and lints the Python (treefmt's
|
|
`ruff-format` and `ruff-check`, which fixes what it can), and the `formatting`
|
|
flake check fails on a file `nix fmt` would change. `nix/vm-tests/ruff.toml`
|
|
names the globals the test driver provides.
|
|
|
|
## Configuration reference
|
|
|
|
`services.hyperhive.deploy.ci-runner.enable = true` on the host that runs it
|
|
(see _For operators_ above). The runner reaches the forge at
|
|
`https://<swarm.forge.domain>`, so that name has to resolve from the host: the
|
|
gateway's dnsmasq answers it locally where the forge runs, and forwards it to
|
|
the operator's DNS everywhere else.
|
|
|
|
Optional tuning under `services.hyperhive.deploy.ci-runner`: `name` (runner
|
|
name in the forge admin panel), `concurrency` (parallel job capacity), `labels`
|
|
(workflow targeting; the default `hive-ci:host` is what this repo's workflows
|
|
name), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration string
|
|
for example `"3h"` — a job that exceeds it's killed so a hung or runaway build
|
|
can't hold the runner's single slot indefinitely).
|
|
|
|
The old `services.hyperhive.deploy.forgejo.ci.{enable,name,concurrency,labels,package,jobTimeout}`
|
|
and `services.hyperhive.swarm.forge.ci.{enable,name,concurrency,labels,package}`
|
|
are still accepted: each forwards to the same-named option under
|
|
`deploy.ci-runner`, and evaluation warns naming both paths. Set the new name to
|
|
silence the warning. The two `name` defaults differ (`hive-ci` before,
|
|
`swarm-ci-runner` now), so a host that never set the old `name` shows up under
|
|
the new default.
|
|
|
|
The forwarded `enable` carries the new module's requirements, not the old
|
|
module's: evaluation refuses a host that sets only the old `enable` unless
|
|
`services.hyperhive.deploy.bao.ciRunnerClientCertFile` and
|
|
`ciRunnerClientKeyFile` are both set. They default to the identity
|
|
`glue-bao-tls.nix` mints when `services.hyperhive.deploy.bao.enable` is true;
|
|
elsewhere place them as described under _For operators_. The runner no longer
|
|
needs `deploy.forgejo.enable` or a hive on its host.
|
|
|
|
## Container design
|
|
|
|
- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI.
|
|
- **Non-ephemeral**: runner credentials (`.runner`) persist across restarts, written to the container's state dir on registration and reused after.
|
|
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/5-project/process/gotchas.md`.
|
|
- **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path. Only `swarm-ci-runner-register.service` loads the registration token; the daemon unit and the jobs it runs have it neither in their environment nor among their credentials.
|
|
|
|
## Registration
|
|
|
|
1. **swarm-controller** (`swarm-controller/src/forge/runner_token.rs`) reads
|
|
the forge's instance-wide token from
|
|
`GET /api/v1/admin/runners/registration-token` at start and every five
|
|
minutes, and writes it to the secret store at
|
|
`swarm/services/ci-runner/registration-token` when it differs from the
|
|
value it last wrote. Its store grant on that path is write-only.
|
|
2. **`swarm-bao-ci-runner.service`** on the runner's host logs in to the store
|
|
with the `ci-runner` leaf, whose `swarm-ci-runner` role reads that one path
|
|
and nothing else, retrying every 30 seconds until the token is there. It
|
|
writes the token to a root-only file in `/run/swarm-ci-runner` on the host,
|
|
which the container sees read-only, and restarts the runner when the value
|
|
changes. `/run` is tmpfs, so no copy reaches persistent disk; after a reboot
|
|
the unit fetches the token again, and until the store answers only a
|
|
runner that has no `.runner` yet waits. It runs at boot, on deploy, and every
|
|
ten minutes on a timer.
|
|
3. **`swarm-ci-runner-register.service`** in the container is the only unit
|
|
that loads that file, with `LoadCredential=`. systemd hands a unit's
|
|
credentials to all of its processes, and for a `host` label act_runner
|
|
runs every job step as a process of the daemon with the daemon's whole
|
|
environment, so the daemon unit carries the token in neither. The
|
|
register unit is a oneshot under the daemon's dynamic user and state
|
|
directory. It reads the token from `$CREDENTIALS_DIRECTORY` and runs
|
|
`act_runner register` only when `.runner` is missing, so the `.runner` it
|
|
writes is the daemon's own. **The runner unit** (`gitea-runner-swarm`)
|
|
starts it in a pre-start step and runs the daemon once it succeeds.
|
|
A root step of the register unit, ahead of registering, removes `.runner`
|
|
in three cases:
|
|
- The forge has no record of the runner. Forgejo answers every call from an
|
|
unknown runner with `unregistered runner`, and the daemon exits 1 when its
|
|
startup `Declare` gets that answer. On the next start the step finds the
|
|
message in the daemon's journal since its last start, and registers again.
|
|
A daemon that's already running when an admin deletes its runner keeps polling
|
|
and logging the error without exiting; the next restart of the unit or
|
|
container registers it again.
|
|
- `.runner` names another forge address than the configured one.
|
|
- `.runner` names other labels than the configured ones, compared in the
|
|
form the daemon writes back: `<name>:<scheme>`, with `host` for a
|
|
label that names no scheme.
|
|
|
|
A changed token alone doesn't register again: the daemon authenticates with
|
|
the per-runner token in `.runner`. Without `.runner` and with no token yet
|
|
the register unit fails and the runner unit's pre-start step with it;
|
|
`Restart=on-failure` on the runner unit starts both again every 15 seconds,
|
|
with no start limit.
|
|
|
|
Host jobs run as the runner's own user, so they can still read `.runner`
|
|
and act as this runner; see the security section below.
|
|
|
|
A new registration is a new runner on the forge: the previous entry stays in
|
|
the forge admin panel as offline until an admin deletes it. The runner the old
|
|
`hive-ci` module registered is one such entry, since this runner registers
|
|
under its own name and token. Delete it under Site administration, Actions,
|
|
Runners.
|
|
|
|
## Actions mirrors
|
|
|
|
<!-- vale write-good.Passive = NO -->
|
|
|
|
With `services.hyperhive.deploy.forgejo.actionsFromSelf` enabled, the swarm
|
|
controller seeds `actions/checkout` and `actions/upload-artifact` pull-mirrors
|
|
(from `github.com/actions/*`) on the forge and Forgejo's `DEFAULT_ACTIONS_URL`
|
|
points at the forge itself. CI `uses: actions/<name>@vN` steps then resolve on
|
|
the forge, with no external DNS on the CI critical path. A workflow that names
|
|
any other bare `actions/*` action fails at job setup with "repository not
|
|
found" until that action joins the list in `nix/host-modules/hive-forge/default.nix`.
|
|
|
|
A forge host that runs the swarm controller therefore resolves those actions
|
|
from itself, with or without a runner on that host.
|
|
Set `actionsFromSelf = false` there to resolve actions from `data.forgejo.org`
|
|
and seed no mirror.
|
|
|
|
The option defaults to whether the forge's host also runs the swarm controller
|
|
(`services.hyperhive.deploy.swarm-controller.enable`; set by `glue-forge-controller.nix`): on when it does, off
|
|
otherwise. Setting it `true` on a forge host without the controller leaves
|
|
the actions unresolved, because nothing seeds the mirrors; evaluation
|
|
warns about it.
|
|
|
|
<!-- vale write-good.Passive = YES -->
|
|
|
|
The **swarm controller** seeds the mirror in its swarm-wide forge-objects
|
|
pass (`swarm-controller/src/forge/objects.rs`), beside the orgs it ensures.
|
|
`glue-forge-controller.nix` forwards the effective mirror list as
|
|
`SWARM_CONTROLLER_FORGE_MIRRORS` in the `swarm-controller` service
|
|
environment (JSON-encoded `[{upstream, dest}]` list). It reads the list of
|
|
the host the controller runs on: mirrors declared on any other host aren't
|
|
seeded, and evaluation warns about them.
|
|
|
|
**The hyperhive mirror**: the same list carries `hyperhive/hyperhive`, from
|
|
`services.hyperhive.deploy.forgejo.hyperhiveUpstream` (default
|
|
`https://forge.darkest.space/hyperhive/hyperhive` on a host that runs the
|
|
controller, unset elsewhere). The controller creates the mirror only when
|
|
`hyperhive/hyperhive` doesn't exist, and it leaves a repo that exists but is
|
|
not a mirror unchanged. `deploy.forgejo.mirrors` takes precedence for the same
|
|
`dest`.
|
|
|
|
**General-purpose mirrors**: you can pre-seed any external repo as a
|
|
pull-mirror via `services.hyperhive.deploy.forgejo.mirrors`:
|
|
|
|
```nix
|
|
services.hyperhive.deploy.forgejo.mirrors = [
|
|
{ upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; }
|
|
{ upstream = "https://github.com/example/tool"; dest = "mirrors/tool"; }
|
|
];
|
|
```
|
|
|
|
The controller creates each entry as a real Forgejo pull-mirror — not a one-off
|
|
clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack`
|
|
request), so a DNS blip during that sync will propagate back to the
|
|
runner as a hard `git clone` failure. The controller
|
|
autocreates the `<owner>` org in `dest`. Keep mirror dests out of the
|
|
hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`)
|
|
to avoid provisioning collisions.
|
|
|
|
## Security: unsandboxed builds and trusted contributors
|
|
|
|
**The runner should only run CI for trusted contributors.** The security boundary is weaker than it looks:
|
|
|
|
### What unsandboxed builds mean
|
|
|
|
nspawn containers can't create user-namespaces, so the module sets `nix.settings.sandbox-fallback = true` in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.
|
|
|
|
A malicious `default.nix` or build script in a PR can therefore:
|
|
|
|
- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at `https://<forge.domain>` (no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic).
|
|
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
|
|
|
|
No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which only `swarm-ci-runner-register.service` loads, so a job step can't read it. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls.
|
|
|
|
Note: `nix flake check --no-build` (eval-only) reduces the exposure but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The CI workflow's `nix flake check` job builds every check derivation, which is the higher-risk path.
|
|
|
|
### Mitigation
|
|
|
|
For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway.
|
|
|
|
For repos with external contributors or fork PRs:
|
|
|
|
- Use Forgejo's **fork PR approval workflow** (`repository.settings` → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
|
|
- Or restrict the CI workflow trigger to push events on branches (not `pull_request` from forks) — forks can't push to upstream branches.
|
|
|
|
The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.
|
|
|
|
## Host store maintenance (recommended)
|
|
|
|
The CI runner builds derivations through the **host** nix-daemon — the
|
|
`swarm-ci-runner` container shares the host store and has no daemon of its own. Build
|
|
outputs accumulate in `/nix/store` with no automatic collection, and a busy
|
|
CI day can fill the disk until every job fails fast with `ENOSPC`.
|
|
|
|
Store GC is a **host-level** concern, so it belongs in the host's own NixOS
|
|
configuration, not in the hyperhive service modules — a single service should
|
|
not reach out and change the host's global nix-daemon options. The module
|
|
only warns at `nixos-rebuild` time when the host sets neither
|
|
`nix.gc.automatic` nor `nix.settings.min-free`. Add the following to your
|
|
host config:
|
|
|
|
```nix
|
|
{
|
|
# Daily GC: delete store paths not referenced by a live root and older
|
|
# than a day. Keeps the store bounded between builds.
|
|
#
|
|
# `randomizedDelaySec` because a fixed daily time lands on the same wall
|
|
# clock as whatever else runs nightly. A collection that overlaps a CI
|
|
# job doesn't fail it — the runner builds through the host nix-daemon,
|
|
# so an in-flight build's temp roots are protected — but the first build
|
|
# after a collection rebuilds what the collection deleted, so a run that
|
|
# lands just behind one pays for it in wall time.
|
|
nix.gc = {
|
|
automatic = true;
|
|
dates = "daily";
|
|
randomizedDelaySec = "45min";
|
|
options = "--delete-older-than 1d";
|
|
};
|
|
|
|
# Disk-pressure GC: when free space drops below min-free mid-build, the
|
|
# daemon collects garbage up to max-free before continuing. This is the
|
|
# real-time net the daily timer can't provide — a same-day build burst is
|
|
# what fills the disk. Tune to your disk size.
|
|
nix.settings.min-free = 20 * 1024 * 1024 * 1024; # 20 GiB
|
|
nix.settings.max-free = 50 * 1024 * 1024 * 1024; # 50 GiB
|
|
}
|
|
```
|
|
|
|
**Measure a collection on your own hosts before relying on that stagger.**
|
|
`randomizedDelaySec` spreads the _start_, not the run, and the sweep over
|
|
unused `.links` entries dominates the run — which scales with the size of the
|
|
store, not with what this collection deletes. On a large store it can take most
|
|
of an hour, so two hosts both set to `daily` may overlap however much jitter you
|
|
give them. `journalctl -u nix-gc.service` reports each run's wall clock.
|
|
|
|
**Remote builders:** if CI dispatches builds to a remote builder (for example via
|
|
`nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_
|
|
store, so apply the same GC config wherever the builder runs —
|
|
GC on the coordinator host won't reclaim space on the builder.
|
|
|
|
⚠️ Two stores also means the temp-root protection above **doesn't span
|
|
them**. Each daemon holds roots only for the builds it runs, so a
|
|
collection on the _builder_ knows nothing about a coordinator build
|
|
waiting on a path there. Stagger the two hosts' GC windows rather than
|
|
giving them the same `dates`, and prefer `min-free`/`max-free` on the
|
|
builder — disk-pressure collection at least runs while nothing is mid-copy.
|
|
|
|
## References
|
|
|
|
- `nix/host-modules/swarm-ci-runner.nix`: runner configuration, token fetch, re-registration step, container setup.
|
|
- `swarm-controller/src/forge/runner_token.rs`: publishes the registration token.
|
|
- `.forgejo/workflows/ci.yml`: workflow definition.
|
|
- `docs/5-project/process/gotchas.md`: nix sandboxing limitations in containers.
|
|
|
|
## Host setup
|
|
|
|
One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that
|
|
sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a
|
|
runner executes pull-request code, so running one is a decision of its own.
|
|
|
|
It needs neither the forge nor a hive on its host. It reaches the forge at
|
|
`https://<swarm.forge.domain>`, and registers with the token swarm-controller
|
|
publishes to the secret store, which it reads under its own `ci-runner` leaf.
|
|
On a host without the store, that leaf is the one file to copy.
|
|
|
|
The runner's built-in Prometheus endpoint is on, bound to `127.0.0.1:9101` in
|
|
the container's own network namespace. The container's journal forwarder
|
|
scrapes it (`services.hyperhive.swarmJournalForwarder.extraScrapeTargets`) and
|
|
pushes the series to the swarm collector under the job `gitea-runner`, so
|
|
nothing listens on the bridge. Series, all prefixed `gitea_runner_`:
|
|
`job_running`, `capacity`, `job_capacity_utilization_ratio`, `uptime_seconds`,
|
|
`info` (labels `version` and `name`), `job_total{status}`,
|
|
`job_duration_seconds`, `poll_fetch_total{result}`,
|
|
`poll_fetch_duration_seconds`, `poll_backoff_seconds`, `report_log_*`,
|
|
`report_state_*` and `client_errors_total`. A pull-request job runs in the same
|
|
namespace and can read the endpoint, which carries counts, durations, the
|
|
runner's name and its version, and no secrets.
|
|
|
|
The **ci runner** board in Grafana shows these series. Its **Runner** filter
|
|
(the series' `service.instance.id`, the scrape address) narrows every panel,
|
|
and uptime, job slots and slot use show one series per runner, so a runner
|
|
registered by hand appears beside the swarm's once its metrics reach the
|
|
store under its own instance id. The forge's **CI runs** tab is the other
|
|
view: run times and queue wait of every run, whichever runner ran it.
|