vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-ci-runner: run the forge CI runner as a swarm service

Browse source 
The CI runner was a hive deploy module: hive-ci.nix ran it beside the
forge, and hive-c0re registered it through the forge's admin API with
the `core` token, writing the registration token through hive-priv. It
could only run on the forge's host, and it kept an admin-scoped forge
token on a hive.

Now the runner is a swarm service:

- swarm-ci-runner.nix runs it in a `swarm-ci-runner` container on the
  host that sets deploy.ci-runner.enable, reaching the forge by name.
  It needs neither the forge nor a hive there.
- swarm-controller reads the forge's runner registration token with its
  own admin token and writes it to swarm/services/ci-runner/
  registration-token whenever it differs from what it last wrote. Its
  grant on that path is create/update only.
- The runner's own leaf (CN swarm-ci-runner, role and policy of the same
  name) reads that one path. swarm-bao-ci-runner.service fetches it into
  the runner's EnvironmentFile and restarts the runner on a change.
- A pre-start step removes .runner when the daemon's journal since its
  last start holds Forgejo's "unregistered runner" refusal (the daemon
  exits 1 when its startup Declare gets it), or when .runner names
  another forge address; upstream's pre-start step then registers again.

Removed from the hive side: hive-ci.nix and deploy.forgejo.ci.*,
forge/ci_runner.rs, PrivRequest::RegisterCiRunner and its handler and
ReadWritePaths entry, InfraContainer::Ci with hivectl's --ci and the
dashboard's hive-ci journal entry, HYPERHIVE_FORGE_CI_ENABLED, and
read:admin,write:admin from CORE_TOKEN_SCOPES: nothing else in hive-c0re
calls /api/v1/admin/*. The old ci options, under both swarm.forge.ci and
deploy.forgejo.ci, now fail evaluation naming deploy.ci-runner.

The forge's half of the old ci.enable, the actions/checkout mirror and
DEFAULT_ACTIONS_URL = self, is deploy.forgejo.actionsFromSelf.

Closes #4954
This commit is contained in:
atlas 2026-10-05 02:25:05 +02:00 • committed by mara
commit e2da5509ae
60 changed files with 1088 additions and 1201 deletions
Download patch file Download diff file Expand all files Collapse all files

2
nix/host-modules/default.nix
View file

@ -17,7 +17,6 @@
./deploy.nix
./local-defaults.nix
./hive-c0re
./hive-ci.nix
./hive-forge
./hive-gateway
./hive-matrix-service.nix
@ -49,6 +48,7 @@
./swarm-bao-service.nix
./swarm-bao.nix
./swarm-ca.nix
./swarm-ci-runner.nix
./swarm-secret-publisher.nix
./swarm-nats-service.nix
./swarm-nats.nix

98
nix/host-modules/deploy.nix
View file

@ -20,9 +20,9 @@
#
# ⚠️ Each entry is an attrset with an `enable`, not a bare bool, so a
# service that grows a second *deployment* decision has somewhere to put
# it — `deploy.forgejo = { enable; ci; }` is then an ordinary addition
# rather than a migration. `ci` ("does this host run the runner too") is
# exactly that shape, and a bare bool leaves it unrepresentable.
# it — `deploy.nats = { enable; autoGenerateCallout; }` is then an ordinary
# addition rather than a migration, where a bare bool leaves it
# unrepresentable.
#
# ⚠️ The renames below are deliberately in this one file rather than
# spread across the service modules, so all of them have a single home
@ -31,6 +31,32 @@
{ lib, config, ... }:
let
deployCfg = config.services.hyperhive.deploy;
# A removed CI-runner option. Its replacement has the same name under
# `deploy.ci-runner`.
ciRunnerRemoved =
from: opt:
lib.mkRemovedOptionModule
(
[
"services"
"hyperhive"
]
++ from
++ [ opt ]
)
''
The CI runner is the swarm service in swarm-ci-runner.nix. It runs on
any host, needs neither the forge nor a hive there, and registers with
the token swarm-controller publishes to the secret store. Set
services.hyperhive.deploy.ci-runner.${opt} instead.
The runner registers as a new runner on the forge, so delete the old
one in the forge's admin panel (Site administration, Actions, Runners).
To keep resolving `uses: actions/checkout` from the forge, set
services.hyperhive.deploy.forgejo.actionsFromSelf = true on its host.
See docs/scheduler/ci.md.
'';
in
{
imports = [
@ -146,33 +172,20 @@ in
[ "services" "hyperhive" "deploy" "swarm-otel" "clientSecretFile" ]
)
# The CI runner, and the only entry here that renames more than an
# `enable`: every knob under it describes the runner THIS host would run,
# so leaving `name`/`concurrency`/`labels`/`package` in the namespace that
# must be identical swarm-wide would keep the original defect for four
# more options. Renamed one by one because `ci` is a plain attrset of
# options rather than a submodule type, so there is no parent path to
# rename in a single entry.
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "forge" "ci" "enable" ]
[ "services" "hyperhive" "deploy" "forgejo" "ci" "enable" ]
)
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "forge" "ci" "name" ]
[ "services" "hyperhive" "deploy" "forgejo" "ci" "name" ]
)
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "forge" "ci" "concurrency" ]
[ "services" "hyperhive" "deploy" "forgejo" "ci" "concurrency" ]
)
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "forge" "ci" "labels" ]
[ "services" "hyperhive" "deploy" "forgejo" "ci" "labels" ]
)
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "forge" "ci" "package" ]
[ "services" "hyperhive" "deploy" "forgejo" "ci" "package" ]
)
# Removed rather than renamed: the runner is a different container with a
# registration of its own, so an old definition is a deployment to re-read
# rather than a path to forward.
(ciRunnerRemoved [ "swarm" "forge" "ci" ] "enable")
(ciRunnerRemoved [ "swarm" "forge" "ci" ] "name")
(ciRunnerRemoved [ "swarm" "forge" "ci" ] "concurrency")
(ciRunnerRemoved [ "swarm" "forge" "ci" ] "labels")
(ciRunnerRemoved [ "swarm" "forge" "ci" ] "package")
(ciRunnerRemoved [ "deploy" "forgejo" "ci" ] "enable")
(ciRunnerRemoved [ "deploy" "forgejo" "ci" ] "name")
(ciRunnerRemoved [ "deploy" "forgejo" "ci" ] "concurrency")
(ciRunnerRemoved [ "deploy" "forgejo" "ci" ] "labels")
(ciRunnerRemoved [ "deploy" "forgejo" "ci" ] "package")
(ciRunnerRemoved [ "deploy" "forgejo" "ci" ] "jobTimeout")
# The queue's callout identity. What stays under `swarm.nats` is what the
# queue IS to every hive — the ports it answers on and the client id it is
@ -216,8 +229,7 @@ in
#
# `mirrors` renames in one entry rather than one per field: it is a single
# option of a list-of-submodule type, so the rename carries its whole
# value. The `ci` block above needs five because it is a plain attrset of
# separate options, which is the case with no parent path to rename.
# value.
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "forge" "openFirewall" ]
[ "services" "hyperhive" "deploy" "forgejo" "openFirewall" ]
@ -451,9 +463,8 @@ in
];
# ⚠️ Only `deploy.forgejo.enable` is declared here. The rest of
# `deploy.forgejo` is in ./hive-forge/default.nix and ./hive-ci.nix: the
# knobs live with the module that reads them; this file stays the registry
# of toggles.
# `deploy.forgejo` is in ./hive-forge/default.nix: the knobs live with the
# module that reads them; this file stays the registry of toggles.
options.services.hyperhive.deploy = {
forgejo.enable = lib.mkOption {
type = lib.types.bool;
@ -602,6 +613,23 @@ in
'';
};
ci-runner.enable = lib.mkOption {
type = lib.types.bool;
default = false;
example = true;
description = ''
Run the swarm's Forgejo Actions runner in a `swarm-ci-runner`
container on this host. It reaches the forge at
{option}`services.hyperhive.swarm.forge.domain` and registers with
the token swarm-controller publishes to the secret store, so it
needs neither the forge nor a hive on this host.
Not derived from
{option}`services.hyperhive.deploy.allSwarmServices`: a runner
executes pull-request code, so running one is a decision of its own.
'';
};
hive-controller.enable = lib.mkOption {
type = lib.types.bool;
default = false;

2
nix/host-modules/glue-bao-readers-policy-order.nix
View file

@ -45,6 +45,8 @@ let
swarm-bao-forwarder-oidc = true;
# ./swarm-nats.nix: the queue's TLS leaf, not a secret, but the same wait.
swarm-bao-nats-tls = deployCfg.nats.enable;
# ./swarm-ci-runner.nix
swarm-bao-ci-runner = deployCfg.ci-runner.enable;
};
# Readers that log in under another unit's principal: reader unit → the

8
nix/host-modules/glue-bao-tls.nix
View file

@ -105,6 +105,8 @@ in
# The store forwarder's.
forwarderOidcClientCertFile = lib.mkDefault "${pkiDir}/forwarder-oidc.pem";
forwarderOidcClientKeyFile = lib.mkDefault "${pkiDir}/forwarder-oidc-key.pem";
ciRunnerClientCertFile = lib.mkDefault "${pkiDir}/ci-runner.pem";
ciRunnerClientKeyFile = lib.mkDefault "${pkiDir}/ci-runner-key.pem";
# The granter: every `swarm-bao-*-policy` unit on this host logs in with
# it. ⚠️ Unlike every other leaf here, never the file an operator copies:
@ -256,6 +258,12 @@ in
[ -s ${pkiDir}/nats-auth.pem ] || ${signLeaf} ${pkiDir} nats-auth \
${lib.escapeShellArg deployCfg.bao.natsAuthCommonName} "" clientAuth
# The CI runner's token fetch. Minted here for the matrix-ctl leaf's
# reason: the runner is a swarm singleton, so on a runner host without
# the store this is the file an operator copies.
[ -s ${pkiDir}/ci-runner.pem ] || ${signLeaf} ${pkiDir} ci-runner \
${lib.escapeShellArg deployCfg.bao.ciRunnerCommonName} "" clientAuth
${lib.optionalString deployCfg.hive-controller.enable ''
# This hive's gateway-leaf identity, which opens
# `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its

4
nix/host-modules/hive-c0re/options.nix
View file

@ -349,8 +349,8 @@
The kernel default is `100`. The hyperhive default of `80` means
agent containers yield slightly to everything that is *not* on
this drop-in path — host services and the infrastructure
containers (`hive-ci`, `hive-forge`, `hive-gateway`,
`hive-matrix`), which stay at `100`. Note this is a hive-wide
containers (`hive-forge`, `hive-gateway`, `hive-matrix`), which
stay at `100`. Note this is a hive-wide
value, so it does not rank agents against *each other*: they all
share one weight.

470
nix/host-modules/hive-ci.nix
View file

@ -1,470 +0,0 @@
{
pkgs,
lib,
config,
...
}:
let
cfg = config.services.hyperhive.deploy.forgejo.ci;
forgeCfg = config.services.hyperhive.swarm.forge;
forgeDeployCfg = config.services.hyperhive.deploy.forgejo;
gatewayCfg = config.services.hyperhive.gateway;
networkCfg = config.services.hyperhive.network;
tlsCfg = config.services.hyperhive.deploy.hive-controller.tls;
# Self-signed TLS is the gateway default (no operator cert / ACME). When
# active, forgejo's ROOT_URL is `https://forge.<domain>` and the leaf is
# signed by the host hive CA — so the runner's Node-based actions (e.g.
# `upload-artifact`, which POSTs to the ROOT_URL-derived artifact endpoint)
# reject the chain, since Node trusts only its bundled CA bundle, not the
# system store. Trust the hive CA explicitly via NODE_EXTRA_CA_CERTS below.
# The bind-mount + `container@` ordering that makes the CA reachable are
# shared with hive-forge via the `hive-ca-trust` helper; only the Node
# `NODE_EXTRA_CA_CERTS` consumption is hive-ci-specific.
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
useSelfSigned = caTrust.useSelfSigned;
caContainerPath = caTrust.caContainerPath;
# The runner daemon needs the hive CA too, and `NODE_EXTRA_CA_CERTS` above
# does not give it to them: that variable is Node's, and it is ADDITIVE.
# `gitea-runner` is Go, whose trust store is REPLACING -- it reads
# `SSL_CERT_FILE` and then trusts only what that file contains. So the two
# consumers need different treatment from the same CA, which is exactly the
# split the helper documents at its head.
#
# Without this the daemon fails every startup call with
# `x509: certificate signed by unknown authority` the moment it reaches a
# TLS endpoint -- and it reaches one whenever the address it is registered
# at redirects to the gateway's https vhost. The unit then crash-loops on
# `Restart=on-failure`, no job is ever picked up, and nothing reports it as
# broken: every pull request simply sits at "Waiting to run".
#
# ⚠️ This is deliberately NOT claimed as the whole story of that outage --
# what address the daemon holds is a separate question. It is here because
# the trust gap is real under every explanation of it, and adding a CA to
# one unit cannot break a path that already works.
caBundleModule = caTrust.trustBundle {
inherit pkgs;
name = "hive-ci";
consumers = [ "gitea-runner-hive" ];
};
# Private network namespace, attached to the hive bridge so the
# runner reaches the forge via the gateway — and cannot reach
# host-loopback (127.0.0.1:7000 dashboard, raw forge port, etc.).
# See docs/networking/network.md.
privateNetwork = true;
in
{
# Forgejo Actions runner in a `hive-ci` nixos-container.
#
# ⚠️ Two properties here are load-bearing and neither is visible from the
# config that implements it:
#
# - The container gets its OWN network namespace, not the host's, so a
# PR's nix build cannot reach host-loopback services. That is a
# prompt-injection boundary; moving this to host netns for convenience
# removes it. Reachability of the forge from inside:
# `docs/networking/network.md::Container isolation`.
# - The forge ADMIN token never enters this container — hive-c0re holds
# it and hands in only a registration token. Full flow:
# `docs/scheduler/ci.md::Container design`.
#
# Non-ephemeral on purpose: gitea-actions-runner writes its registered
# credentials to its stateDir on first registration and reuses them, so an
# ephemeral container would re-register on every restart.
#
# Nix builds here use the shared /nix/store with `sandbox-fallback = true`
# — nspawn containers cannot create the user namespaces nix sandboxing
# wants: `docs/process/gotchas.md`.
options.services.hyperhive.deploy.forgejo.ci = {
enable = lib.mkOption {
type = lib.types.bool;
default = false;
example = true;
description = ''
Run a Forgejo Actions runner in a `hive-ci` nixos-container.
Grouped under `services.hyperhive.deploy.forgejo` because the runner
is tightly coupled to the forge instance it registers against.
Disabled by default. It registers against the forge on this same
host, so it requires
{option}`services.hyperhive.deploy.forgejo.enable` here.
On first start the container auto-registers against hive-forge using
hive-c0re's admin token — no manual token provisioning needed.
Runner credentials are persisted in the container's state dir and
reused on every subsequent boot.
'';
};
name = lib.mkOption {
type = lib.types.str;
default = "hive-ci";
example = "prod-hive";
description = ''
Runner name as shown in the Forgejo admin panel. Defaults to
"hive-ci"; override when multiple hives share a Forgejo instance.
'';
};
concurrency = lib.mkOption {
type = lib.types.ints.positive;
default = 1;
example = 4;
description = ''
Maximum number of workflow jobs the runner executes in parallel.
Each job gets its own temporary working directory; multiple parallel
jobs share the container's nix store and cargo registry cache.
Higher values trade memory + CPU headroom for throughput.
'';
};
labels = lib.mkOption {
type = lib.types.listOf lib.types.str;
default = [ "hive-ci:host" ];
example = [
"hive-ci:host"
"nix:host"
];
description = ''
Runner labels in `<name>:<scheme>` format. The `host` scheme runs
commands directly in the container (no docker/podman). Workflow
files target this runner with `runs-on: [hive-ci]`.
'';
};
package = lib.mkOption {
type = lib.types.package;
default = pkgs.gitea-actions-runner;
defaultText = lib.literalExpression "pkgs.gitea-actions-runner";
description = "gitea-actions-runner package.";
};
jobTimeout = lib.mkOption {
type = lib.types.str;
default = "1h";
example = "3h";
description = ''
Per-job wall-clock timeout the runner enforces (act_runner's
`runner.timeout`). A job that exceeds it is killed, so a hung or
runaway build is bounded instead of holding the runner's single
slot indefinitely. Default `1h` comfortably covers a cold-cache
nix build while still bounding a stuck job; raise it (e.g.
`"3h"`) if you legitimately run jobs longer than that. Accepts a
Go duration string (`30m`, `1h`, `2h30m`). Note: this is
enforced by the runner process, so it only fires while that
process is itself healthy.
'';
};
};
config = lib.mkIf cfg.enable {
assertions = [
{
# The runner reaches the forge through THIS host's gateway, and
# hive-c0re registers it through the local forge container; neither
# exists on a host that does not run the forge. A runner on another
# host, registered by the swarm instead, is a separate design.
assertion = forgeDeployCfg.enable;
message = ''
services.hyperhive.deploy.forgejo.ci.enable requires
services.hyperhive.deploy.forgejo.enable on the same host.
The CI runner reaches the forge through this host's gateway and
is registered through the local forge container, so it can only
run where the forge does. Enable CI on the swarm's forge host.
'';
}
];
# The runner container has a private netns of its own, so it is
# attached to the bridge and reaches the forge by name through the
# gateway vhost the assertion above already insists on.
services.hyperhive.network.enable = lib.mkDefault true;
services.hyperhive.gateway.enable = lib.mkDefault true;
services.hyperhive.gateway.dns.enable = lib.mkDefault true;
# Create /run/hive-ci/ on the host and seed runner-token with a
# placeholder. The container bind-mounts this file read-only; hive-c0re
# (via hive-priv's RegisterCiRunner) overwrites it with the real
# registration token when it registers the runner out of band. The
# placeholder keeps the runner's EnvironmentFile present from first boot,
# before c0re has registered — the runner's ExecStartPre precond (below)
# distinguishes the placeholder from a real token.
systemd.tmpfiles.rules = [
"d /run/hive-ci 0700 root root -"
"f /run/hive-ci/runner-token 0600 root root - TOKEN=placeholder"
];
# Tell hive-c0re that CI is enabled so its startup sweep registers the
# runner. Registration is OFF the container's boot-critical path:
# hive-c0re (`forge/ci_runner.rs`) validates the persisted `.runner`
# against the forge and, when absent/stale, mints a registration token and
# hands it to hive-priv to write `/run/hive-ci/runner-token` + restart the
# runner. The forge admin token stays in hive-c0re; only the registration
# token reaches the host env-file the container mounts read-only.
systemd.services.hive-c0re.environment.HYPERHIVE_FORGE_CI_ENABLED = "1";
# `caTrust.containerOrdering` orders this unit after `hive-tls-ca.service`
# in self-signed mode (see the hive-ca-trust helper). Runner registration
# is not on the boot-critical path — hive-c0re owns it out of band — so
# the container boots immediately and needs no start-timeout bump.
systemd.services."container@hive-ci" = caTrust.containerOrdering;
containers.hive-ci = {
autoStart = true;
ephemeral = false;
# Journal files on the host, not inside the container: nixpkgs hardcodes
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it.
extraFlags = [ "--link-journal=host" ];
inherit privateNetwork;
hostBridge = networkCfg.bridgeName;
bindMounts = {
# Pre-filled by hive-c0re (via hive-priv) with the runner
# registration token; tmpfiles seeds a `TOKEN=placeholder` before
# that. Read-only: the container reads TOKEN= from here; the core
# admin token never enters the container.
"/run/hive-ci/runner-token" = {
hostPath = "/run/hive-ci/runner-token";
isReadOnly = true;
};
# Route the container's nix through the HOST nix-daemon (as the agent
# containers do) instead of an in-container daemon: with
# NIX_REMOTE=daemon below, builds run on the host daemon and inherit
# its buildMachines + max-jobs. Note that `fallback` is NOT inherited
# that way — it's a client-side option transmitted per connection, so
# the container sets it itself (see nix.settings.fallback below).
# Bind the *directory* (not the socket file) so the mount stays live
# across a host nix-daemon restart, which recreates the socket inode.
"/nix/var/nix/daemon-socket" = {
hostPath = "/nix/var/nix/daemon-socket";
isReadOnly = false;
};
}
# Self-signed mode: bind the public hive CA cert read-only so the
# runner's Node actions trust the gateway/forge leaf (consumed via
# NODE_EXTRA_CA_CERTS in the container config). Shared bind-mount +
# ordering come from the hive-ca-trust helper.
// caTrust.bindMount;
config =
{ pkgs, lib, ... }:
{
# Assembles system CAs + the hive CA into one bundle and sets
# `SSL_CERT_FILE` on the runner unit. See `caBundleModule` above for
# why the Node variable beside it is not enough.
imports = [
../container-modules/swarm-container.nix
caBundleModule
];
# Its own netns, so it keeps its own firewall; resolvconf stays on
# and serves `networking.nameservers` below.
services.hyperhive.swarmContainer = {
inherit privateNetwork;
writesOwnResolvConf = false;
};
# Point the forge domain at the bridge IP so the runner can
# reach the forge through the gateway — both for registration /
# polling (runner URL below) and for artifact uploads (the
# Forgejo Actions artifact API uses ROOT_URL, i.e. the public
# forge domain, not a localhost URL). The gateway vhost for
# `forgeCfg.domain` proxies all `/` → forge; `addSSL = true`
# means HTTP:80 is served without redirect alongside HTTPS:443.
# Ports 80 and 443 are always open on the bridge firewall (see
# hive-network.nix). No DNS lookup needed — /etc/hosts wins.
networking.extraHosts = "${networkCfg.bridgeIp} ${forgeCfg.domain}";
# DNS: use the hive resolver on the bridge IP (dnsmasq in
# hive-gateway) for external lookups (git checkout, crate
# registries, etc.). The bridge→loopback DROP rule does not
# affect traffic destined for the bridge IP itself.
networking.nameservers = [ networkCfg.bridgeIp ];
# Bridge-attached via privateNetwork=true + hostBridge. The
# gateway's dnsmasq serves a DHCP pool covering all usable bridge
# addresses (see dhcp-range in hive-gateway.nix) — agents and
# service containers alike receive IPs dynamically.
networking.interfaces.eth0.useDHCP = true;
# nspawn containers can't create user-namespaces, so nix
# sandboxing always fails. Fall back to unsandboxed builds.
# Moot once every nix invocation in the container routes
# through the host daemon (the daemon governs sandboxing).
# See docs/process/gotchas.md and nix/agent-modules/default.nix.
nix.settings.sandbox-fallback = lib.mkForce true;
# Degrade to a local build when a remote builder is unreachable
# rather than failing the check. `fallback` is a client-side
# option (transmitted per daemon connection), so routing through
# the host daemon does NOT inherit the host's value — the
# container has to set it. Same reasoning as the agent
# containers; see nix/agent-modules/default.nix.
nix.settings.fallback = true;
nix.settings.experimental-features = [
"nix-command"
"flakes"
];
# Build through the HOST nix-daemon (bind-mounted socket above), not
# an in-container daemon. NIX_REMOTE=daemon makes root services (the
# runner) use the daemon socket rather than store=auto → local store.
# Disable BOTH the service and its .socket unit: the .socket owns the
# listen path /nix/var/nix/daemon-socket/socket and would otherwise
# race the bind mount over the same path.
systemd.globalEnvironment.NIX_REMOTE = "daemon";
systemd.services.nix-daemon.enable = false;
systemd.sockets.nix-daemon.enable = false;
# package is top-level on gitea-actions-runner, not per-instance.
services.gitea-actions-runner.package = cfg.package;
services.gitea-actions-runner.instances.hive = {
enable = true;
name = cfg.name;
# Route through the gateway (bridge IP, port 80) so the
# runner never touches host-loopback. The forge domain
# resolves to the bridge IP via networking.extraHosts above;
# the gateway vhost `forgeCfg.domain` proxies to the forge
# on HTTP:80 (addSSL=true, no HTTP→HTTPS redirect).
url = "http://${forgeCfg.domain}";
# EnvironmentFile providing TOKEN= — pre-filled by hive-c0re
# (via hive-priv) with the runner registration token;
# bind-mounted read-only from /run/hive-ci/runner-token on the
# host. The runner's ExecStartPre precond gates on this being a
# real (non-placeholder) token or an already-registered .runner.
tokenFile = "/run/hive-ci/runner-token";
labels = cfg.labels;
settings = {
runner.capacity = cfg.concurrency;
# Per-job wall-clock cap — see the `jobTimeout` option.
runner.timeout = cfg.jobTimeout;
};
};
# No tmpfiles rule: /run/hive-ci/runner-token is bind-mounted
# read-only from the host (pre-filled before container start).
# nspawn creates the /run/hive-ci/ mount-point directory
# automatically before launching the container's init.
# No hive-ci-register.service inside the container: all forge
# API calls (runner validation, token fetch) live in hive-c0re
# (forge/ci_runner.rs), out of band. The core admin token never
# enters this container.
# git is already in the gitea-actions-runner service PATH (the
# nixpkgs module builds it from the package's runtime deps).
# nix is required for `nix flake check` / `nix build` workflow
# steps — it's not included by the upstream module.
# Use the `path` service attribute (generates ExecSearchPath=)
# to prepend nix's bin dir to PATH without touching the
# environment.PATH the nixpkgs module sets — overriding that
# would lose git, curl, nodejs, and other runner deps.
environment.systemPackages = [
pkgs.git
pkgs.nix
];
systemd.services."gitea-runner-hive" = {
path = [ pkgs.nix ];
# A CI runner must be able to EXECUTE what it builds, and by
# default it cannot: the upstream unit uses `DynamicUser`, and
# systemd mounts a dynamic unit's state directory `noexec`.
# Measured on the live runner rather than inferred — from a job's
# own /proc/self/mountinfo:
#
# /var/lib/private ro,nosuid,nodev,noexec (tmpfs)
# /var/lib/private/gitea-runner rw,nosuid,nodev,noexec,idmapped
#
# with /tmp as the control at rw,nosuid,nodev — no noexec, and a
# script there executes fine. Nothing noticed for as long as every
# workflow built inside the nix sandbox and executed nothing in the
# workspace; the first job that compiled a build script into the
# runner's own target dir died with EACCES on execve, reported as
# "could not execute process ... (never executed)".
#
# Both spellings are listed because `ExecPaths=` resolves against
# the HOST root, where the dynamic-user layout makes
# /var/lib/gitea-runner a symlink into private/. The `-` prefix
# means "ignore if absent", so whichever layout the unit ends up
# with, the other is a no-op rather than a startup failure.
serviceConfig.ExecPaths = [
"-/var/lib/gitea-runner"
"-/var/lib/private/gitea-runner"
];
# Trust the hive CA in Node-based actions. With self-signed TLS,
# forgejo's ROOT_URL is `https://forge.<domain>` (CA-signed leaf),
# so actions like `upload-artifact` — whose Node HTTP client uses
# Node's *bundled* CA bundle, not the system store — reject the
# chain with "unable to verify the first certificate". Pointing
# NODE_EXTRA_CA_CERTS at the bind-mounted CA adds it to Node's
# roots for every action, hive-wide. Inherited by the job
# processes the runner spawns (host execution mode). Only set in
# self-signed mode; with an operator cert / ACME the public CA
# already validates and the bind-mount is absent.
environment = lib.mkIf useSelfSigned {
NODE_EXTRA_CA_CERTS = caContainerPath;
};
# Gate runner start on credentials being present. Registration is
# done OUT OF BAND by hive-c0re (forge/ci_runner.rs): it mints a
# token and, via hive-priv, writes /run/hive-ci/runner-token +
# restarts this unit. Pass iff already registered (.runner present)
# OR a real (non-placeholder) token is in place; else exit 1 so the
# runner stays down (Restart=on-failure retries) until c0re
# registers it, rather than blocking the container's start.
#
# No nix-daemon wait: builds route through the HOST daemon
# (bound socket above), which is up before the container — nix's
# own connect retries cover any brief window. `mkBefore` runs this
# ahead of any upstream pre-steps.
#
# The `+` prefix runs this step with FULL PRIVILEGES instead of as
# the unit's (dynamic) `gitea-runner` user. The token file is 0600
# root-owned, so an unprivileged check cannot read it — and it did
# not fail closed: `[ -s ]` succeeds on a stat alone, `grep` then
# died with EACCES, and `! grep` turned that error into `true`, so
# the gate exited 0 on an unreadable file. It reported "a real token
# is present" without ever having looked, for its entire existence.
#
# This grants the runner nothing new: `tokenFile` above becomes
# `EnvironmentFile=`, which systemd already reads AS ROOT before
# dropping privileges, so the payload never passes through an
# unprivileged reader either way. The `+` only lets the gate observe
# what systemd observes.
serviceConfig.ExecStartPre = lib.mkBefore [
"+${pkgs.writeShellScript "hive-ci-runner-precond" ''
RUNNER=/var/lib/gitea-runner/hive/.runner
TOKEN_FILE=/run/hive-ci/runner-token
if [ -f "$RUNNER" ]; then exit 0; fi
# Readability is asserted SEPARATELY and fails loudly. Folding
# it into the test below is what hid the bug: an unreadable file
# and a file holding a real token produced the same verdict, so
# "I could not look" was indistinguishable from "I looked and it
# is fine".
if [ -e "$TOKEN_FILE" ] && [ ! -r "$TOKEN_FILE" ]; then
echo "hive-ci runner: $TOKEN_FILE exists but is unreadable by this step; the credential gate cannot run" >&2
exit 1
fi
if [ -s "$TOKEN_FILE" ] && ! ${pkgs.gnugrep}/bin/grep -q '^TOKEN=placeholder$' "$TOKEN_FILE"; then
exit 0
fi
echo "hive-ci runner: not registered and no real token yet; waiting for hive-c0re to register" >&2
exit 1
''}"
];
# Registration is out of band: hive-c0re explicitly restarts this
# unit once it writes the real token, which is the primary path. As
# a safety net, if the precond above fires first (no creds yet at
# boot) retry until c0re registers, rather than staying down — with
# no start-limit rate cap so it keeps retrying however long forge /
# the core token take to settle.
serviceConfig.Restart = lib.mkForce "on-failure";
# `mkForce`: the upstream gitea-actions-runner module also sets
# RestartSec (2), so ours needs the higher priority or the two
# collide into an eval error — same as the sibling `Restart`.
serviceConfig.RestartSec = lib.mkForce 15;
startLimitIntervalSec = 0;
};
};
};
};
}

49
nix/host-modules/hive-forge/default.nix
View file

@ -74,23 +74,18 @@ let
"https://${cfg.domain}${portSuffix}/";
effectiveRootUrl = if cfg.rootUrl != null then cfg.rootUrl else defaultRootUrl;
# When CI is enabled, the runner needs `actions/checkout` resolvable
# without external DNS (hive-ci shares the host netns, so a host-resolver
# blip otherwise reds every `actions/checkout@vN` fetch from
# data.forgejo.org). Auto-append a pull-mirror of it and point
# forgejo's DEFAULT_ACTIONS_URL at this instance so `uses:` resolves local.
ciEnabled = config.services.hyperhive.deploy.forgejo.ci.enable;
# `actionsFromSelf` resolves `uses:` against this forge, which then has to
# hold the action.
actionsFromSelf = deployCfg.forgejo.actionsFromSelf;
actionCheckoutMirror = {
upstream = "https://github.com/actions/checkout";
dest = "actions/checkout";
};
# Auto-append the actions/checkout mirror only when CI is on AND the
# operator hasn't already declared that dest themselves (else CI-on +
# an explicit `actions/checkout` entry would duplicate it).
# Unless the operator already declared that dest, which would duplicate it.
effectiveMirrors =
deployCfg.forgejo.mirrors
++ lib.optional (
ciEnabled && !(lib.any (m: m.dest == actionCheckoutMirror.dest) deployCfg.forgejo.mirrors)
actionsFromSelf && !(lib.any (m: m.dest == actionCheckoutMirror.dest) deployCfg.forgejo.mirrors)
) actionCheckoutMirror;
# Share host netns — forgejo's HTTP / SSH listeners then look
@ -179,12 +174,24 @@ in
so a host-resolver blip leaves a *stale* mirror, never a hard
failure on whatever reads it.
When `services.hyperhive.deploy.forgejo.ci.enable` is set, an
`actions/checkout` mirror is auto-appended to this list and
forgejo's `DEFAULT_ACTIONS_URL` is pointed at this instance, so CI
`uses: actions/checkout@vN` steps resolve entirely on loopback with
no external DNS on the critical path (the seed/re-sync needs
external DNS, but that's off the CI path).
With {option}`services.hyperhive.deploy.forgejo.actionsFromSelf`
set, an `actions/checkout` mirror is appended to this list.
'';
};
actionsFromSelf = lib.mkOption {
type = lib.types.bool;
default = false;
example = true;
description = ''
Resolve a workflow's `uses: <org>/<action>@vN` against this forge
instead of `data.forgejo.org`, and add an `actions/checkout`
pull-mirror to {option}`services.hyperhive.deploy.forgejo.mirrors`
so the checkout step has something to resolve to. A CI job's
checkout then needs no external DNS; only the mirror's re-sync
does.
Any other action a workflow names has to be mirrored here too.
'';
};
@ -641,16 +648,16 @@ in
# Enable Forgejo Actions so the runner registration token
# API endpoint is available. Without this the endpoint
# returns "runner registration token not found" regardless
# of token scopes. Required by `hive-ci-register.service`
# in the hive-ci container.
# of token scopes. swarm-controller's `forge::runner_token`
# reads it.
actions.ENABLED = true;
# When CI is enabled, resolve `uses: <org>/<action>@vN` from
# With `actionsFromSelf`, resolve `uses: <org>/<action>@vN` from
# THIS instance (the seeded `actions/checkout` pull-mirror)
# instead of the upstream default `data.forgejo.org` — keeps
# the checkout step on loopback, immune to a host-resolver
# blip. `self` = forgejo expands actions against its
# own ROOT_URL.
actions.DEFAULT_ACTIONS_URL = lib.mkIf ciEnabled "self";
actions.DEFAULT_ACTIONS_URL = lib.mkIf actionsFromSelf "self";
# F3 (federation) computes its data dir relative to the
# forgejo binary, which lands in the read-only nix
# store and crashes anything that touches the F3
@ -1099,7 +1106,7 @@ in
lib.mkIf deployCfg.swarm-controller.enable (builtins.toJSON effectiveMirrors);
warnings = lib.optional (effectiveMirrors != [ ] && !deployCfg.swarm-controller.enable) ''
services.hyperhive.deploy.forgejo.mirrors (or the actions/checkout mirror
deploy.forgejo.ci.enable adds) is set on a host that does not run the
deploy.forgejo.actionsFromSelf adds) is set on a host that does not run the
swarm-controller. The controller seeds mirrors from its own host's list,
so these are not seeded: declare them on the swarm-controller's host.
'';

2
nix/host-modules/hive-gateway/dnsmasq.nix
View file

@ -68,7 +68,7 @@
# DHCP pool covering all usable host addresses on the bridge
# subnet — bounds computed by hive-network.nix from
# bridgeIp/bridgePrefixLength. All containers (agents and service
# containers such as hive-ci) receive their IPs dynamically.
# containers such as swarm-ci-runner) receive their IPs dynamically.
dhcp-range = "${networkCfg.dhcpRangeStart},${networkCfg.dhcpRangeEnd},1h";
dhcp-leasefile = "/var/lib/dnsmasq/dnsmasq.leases";
# No explicit upstream: non-hive queries follow dnsmasq's

2
nix/host-modules/hive-gateway/options.nix
View file

@ -229,7 +229,7 @@ in
Read-only derived flag: `true` when the gateway serves the
self-signed leaf the secret store issues — i.e. neither `tls.certDir` nor
`tls.acme.enable` is configured. Single source of truth for the
self-signed condition; consumed by the `hive-tls` and `hive-ci`
self-signed condition; consumed by the `hive-tls` and `hive-ca-trust`
modules so the derivation isn't duplicated. Internal — not meant to
be set by operators (use `tls.certDir` / `tls.acme` to override the
self-signed default).

2
nix/host-modules/hive-matrix.nix
View file

@ -11,7 +11,7 @@ let
gatewayCfg = config.services.hyperhive.gateway;
hyperhiveDomain = config.services.hyperhive.domain;
# Same runtime→build-time bridge hive-ci and hive-forge already cross:
# Same runtime→build-time bridge hive-forge and swarm-ci-runner cross:
# binds the hive trust bundle (which folds in the swarm root) into the
# container and orders the container after `hive-tls-ca.service`. The
# assembled bundle itself comes from `caTrust.trustBundle`, imported in

2
nix/host-modules/hive-network.nix
View file

@ -42,7 +42,7 @@ let
d.swarm-controller.enable
d.swarm-ui.enable
d.forgejo.enable
d.forgejo.ci.enable
d.ci-runner.enable
d.matrix.enable
d.authelia.enable
d.bao.enable

4
nix/host-modules/hive-priv.nix
View file

@ -118,9 +118,6 @@ in
# /run/hive-agent — creates per-agent socket dirs
# /run/systemd — container@ drop-ins + machined state
# /run/lock — nixos-container's create/destroy lock file
# /run/hive-ci — register_ci_runner's runner-token write
# (else token stays placeholder → runner
# crash-loops → no CI hive-wide)
# /var/lib/nixos-containers — container rootfs
# /var/lib/hyperhive — agent state (forge/matrix token files)
# /nix — nix store + profile updates on create/update
@ -131,7 +128,6 @@ in
"/run/hive-agent"
"/run/systemd"
"/run/lock"
"/run/hive-ci"
"/var/lib/nixos-containers"
"/var/lib/hyperhive"
"/nix"

8
nix/host-modules/lib/hive-ca-trust.nix
View file

@ -9,7 +9,7 @@
#
# `bindMount` + `containerOrdering` are the language-agnostic half. The
# *consumption* differs per runtime: an additive variable (Node's
# `NODE_EXTRA_CA_CERTS`, hive-ci) points straight at `caContainerPath` from
# `NODE_EXTRA_CA_CERTS`, swarm-ci-runner) points straight at `caContainerPath` from
# the call site, while a *replacing* one (Go's `SSL_CERT_FILE`, rustls) needs
# the system-CAs+hive-CA concat that `trustBundle` below does for it.
#
@ -19,7 +19,7 @@
# caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
# # then, in the container:
# # bindMounts = { … } // caTrust.bindMount;
# # systemd.services."container@hive-ci" = lib.mkMerge [ caTrust.containerOrdering … ];
# # systemd.services."container@swarm-ci-runner" = lib.mkMerge [ caTrust.containerOrdering … ];
# # environment.NODE_EXTRA_CA_CERTS = caTrust.caContainerPath; # consumption, per-caller
#
# `tlsCfg` = config.services.hyperhive.deploy.hive-controller.tls
@ -87,7 +87,7 @@ in
};
# Fold into the caller's `container@<name>` unit (via `lib.mkMerge` if the
# caller adds its own keys, e.g. hive-ci's `TimeoutStartSec`). Orders the
# caller adds its own keys). Orders the
# container after the host `hive-tls-ca.service` so the bind source exists
# before nspawn sets the mount up — a condition-skipped/late CA would
# otherwise fail the container start.
@ -99,7 +99,7 @@ in
# System CAs + hive CA in one bundle, with `SSL_CERT_FILE` set on each
# consumer — for runtimes whose trust variable *replaces* the store (Go,
# rustls-native-certs). An additive one (Node's `NODE_EXTRA_CA_CERTS`,
# hive-ci) needs no bundle and should not use this.
# swarm-ci-runner) needs no bundle and should not use this.
#
# imports = [ (caTrust.trustBundle { inherit pkgs; name = "swarm-nats";
# consumers = [ "swarm-nats-auth" ]; }) ];

62
nix/host-modules/swarm-bao.nix
View file

@ -429,6 +429,10 @@ let
capabilities = ["create", "read", "update"]
}
path "${credentialMountPath}/data/${ciRunnerTokenLeaf}" {
capabilities = ["create", "update"]
}
path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" {
capabilities = ["read"]
}
@ -519,6 +523,12 @@ let
# it; the controller reads it and holds no other copy.
controllerClientLeaf = "swarm/controller/swarm-controller/oidc/client";
# The forge's runner registration token, the nix half of
# `swarm_secret_client::ci::registration_token_path`. The controller writes
# it and cannot read it back (`forge::runner_token` keeps the value it last
# wrote in memory); the CI runner's fetch unit is its one reader.
ciRunnerTokenLeaf = "swarm/services/ci-runner/registration-token";
# The UI's OIDC client secret. The publisher writes it; the granter reads it
# into the `oidc` auth method's config, which is the only copy bao uses.
baoUiClientLeaf = "swarm/services/${cfg.ui.oidc.clientId}/oidc/client";
@ -853,6 +863,16 @@ let
}
];
# The CI runner's token fetch. The one reader of the path, which the
# controller writes and cannot read back.
ciRunnerReaders = [
{
name = "swarm-ci-runner";
cn = baoDeploy.ciRunnerCommonName;
policyText = readStanza "${credentialMountPath}/data/${ciRunnerTokenLeaf}";
}
];
# The role name IS the policy name, as for the three service principals
# above: the role attaches the policy by spelling it identically, and one
# string for both objects removes the way they drift apart.
@ -1640,6 +1660,19 @@ in
'';
};
ciRunnerCommonName = lib.mkOption {
type = lib.types.str;
default = "swarm-ci-runner";
description = ''
Subject the store's `swarm-ci-runner` cert-auth role accepts: the
identity `swarm-bao-ci-runner`, the unit that fetches the forge's
runner registration token for the CI runner, presents. Its grant is one
path, `swarm/services/ci-runner/registration-token`, and read only.
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
'';
};
queueAgentCommonNamePrefix = lib.mkOption {
type = lib.types.str;
default = "swarm-bao-queue-agent";
@ -1847,6 +1880,32 @@ in
'';
};
ciRunnerClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/ci-runner.pem";
description = ''
Certificate the unit that fetches the CI runner's registration token
presents to the store. Its subject must be
{option}`services.hyperhive.deploy.bao.ciRunnerCommonName`.
./glue-bao-tls.nix points it at the leaf it mints on the store's host.
On a runner host without the store, it is a file the operator copies
from the store's host. A path, never a value.
'';
};
ciRunnerClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/ci-runner-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.bao.ciRunnerClientCertFile`.
A path, never a value.
'';
};
granterCommonName = lib.mkOption {
type = lib.types.str;
default = "bao-granter";
@ -2907,6 +2966,8 @@ in
systemd.services.swarm-bao-nats-auth-policy = readerPolicyUnit "write the queue responder's agent-credential-reader bao policy and cert-auth role" natsAuthReaders;
systemd.services.swarm-bao-ci-runner-policy = readerPolicyUnit "write the CI runner's registration-token-reader bao policy and cert-auth role" ciRunnerReaders;
# A FOURTH sibling, same shape and same reasons as the two above. This
# one is what turns `swarm-services-issuer` from a declaration into a
# grant: a bao policy reaches nothing until a login role hands it to a
@ -3111,6 +3172,7 @@ in
"swarm-bao-services-issuer-policy.service"
"swarm-bao-nats-tls-policy.service"
"swarm-bao-hive-leaf-policy.service"
"swarm-bao-ci-runner-policy.service"
]
++ granterAfter;
requires = [ "swarm-bao-pki.service" ];

421
nix/host-modules/swarm-ci-runner.nix Normal file
View file

@ -0,0 +1,421 @@
# The swarm's Forgejo Actions runner, in a `swarm-ci-runner` container on the
# host that sets `deploy.ci-runner.enable`. It reaches the forge by name and
# reads its registration token from the secret store under a certificate of
# its own, so nothing here asks whether this host runs the forge or a hive.
#
# ⚠️ Two properties here are load-bearing and neither is visible from the
# config that implements it:
#
# - The container gets its OWN network namespace, not the host's, so a PR's
# nix build cannot reach host-loopback services. That is a
# prompt-injection boundary; moving this to host netns for convenience
# removes it. `docs/networking/network.md::Container isolation`.
# - The forge ADMIN token never enters this container. swarm-controller holds
# it and publishes only the registration token, which the runner's own store
# identity reads. `docs/scheduler/ci.md::Registration`.
#
# Non-ephemeral on purpose: the runner writes its registered credentials
# (`.runner`) to its state dir and reuses them, so an ephemeral container
# would register a new runner on every restart.
{
pkgs,
lib,
config,
...
}:
let
cfg = config.services.hyperhive.deploy.ci-runner;
forgeCfg = config.services.hyperhive.swarm.forge;
baoCfg = config.services.hyperhive.swarm.bao;
baoDeploy = config.services.hyperhive.deploy.bao;
gatewayCfg = config.services.hyperhive.gateway;
networkCfg = config.services.hyperhive.network;
tlsCfg = config.services.hyperhive.deploy.hive-controller.tls;
machine = "swarm-ci-runner";
# The upstream module names its unit `gitea-runner-<instance>`.
instance = "swarm";
runnerUnit = "gitea-runner-${instance}";
# What the runner registers against and records in `.runner`. The precond
# below compares the two, so both come from here.
forgeUrl = "https://${forgeCfg.domain}";
# The forge's gateway serves a leaf from the swarm CA in self-signed mode.
# The runner daemon is Go, whose `SSL_CERT_FILE` replaces the system store,
# so it gets the assembled bundle; the Node actions it runs get
# `NODE_EXTRA_CA_CERTS`, which is additive, further down.
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
caBundleModule = caTrust.trustBundle {
inherit pkgs;
name = machine;
consumers = [ runnerUnit ];
};
# The upstream unit is `DynamicUser`, so its state dir is
# `/var/lib/private/gitea-runner`; `/var/lib/gitea-runner` is a symlink to
# it, which is the spelling a step run as root (`+`) can follow.
runnerFile = "/var/lib/gitea-runner/${instance}/.runner";
# Inside the container, root-only. The token file is the runner's
# `EnvironmentFile` (`TOKEN=<value>`); the host unit below writes it through
# the container's root, at `hostTokenFile`.
stateDir = "/var/lib/swarm-ci-runner";
tokenFile = "${stateDir}/registration-token.env";
hostTokenFile = "/var/lib/nixos-containers/${machine}${tokenFile}";
# The epoch second the daemon last started. The precond reads the daemon's
# journal from there.
daemonStartedFile = "${stateDir}/daemon-started";
# The nix half of `swarm_secret_client::ci::registration_token_path`, in the
# form `bao kv get` takes (no KV v2 `data/`).
storePath = "secret/swarm/services/ci-runner/registration-token";
# What Forgejo answers a runner it has no record of — deleted in the admin
# panel, or a forge whose database was replaced — on every call but
# `Register` (`routers/api/actions/runner/interceptor.go`). The daemon logs
# it and exits 1 when its startup `Declare` gets it. ⚠️ Nothing this module
# prints may contain it: the precond would read its own line as a rejection.
rejectedMessage = "unregistered runner";
haveClientIdentity =
baoDeploy.ciRunnerClientCertFile != null && baoDeploy.ciRunnerClientKeyFile != null;
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
refreshConsumer = import ./lib/refresh-consumer.nix { };
storeRetry = import ./lib/store-retry.nix { };
privateNetwork = true;
in
{
options.services.hyperhive.deploy.ci-runner = {
# `enable` is in ./deploy.nix.
name = lib.mkOption {
type = lib.types.str;
default = "swarm-ci-runner";
description = ''
Runner name as shown in the Forgejo admin panel.
'';
};
concurrency = lib.mkOption {
type = lib.types.ints.positive;
default = 1;
example = 4;
description = ''
Maximum number of workflow jobs the runner executes in parallel.
Each job gets its own temporary working directory; parallel jobs
share the host's nix store and the container's cargo registry
cache. Higher values trade memory and CPU headroom for throughput.
'';
};
labels = lib.mkOption {
type = lib.types.listOf lib.types.str;
default = [ "hive-ci:host" ];
example = [
"hive-ci:host"
"nix:host"
];
description = ''
Runner labels in `<name>:<scheme>` format. The `host` scheme runs
commands directly in the container (no docker/podman). This repo's
workflows target the default with `runs-on: hive-ci`.
'';
};
package = lib.mkOption {
type = lib.types.package;
default = pkgs.gitea-actions-runner;
defaultText = lib.literalExpression "pkgs.gitea-actions-runner";
description = "gitea-actions-runner package.";
};
jobTimeout = lib.mkOption {
type = lib.types.str;
default = "1h";
example = "3h";
description = ''
Per-job wall-clock timeout the runner enforces (act_runner's
`runner.timeout`). A job that exceeds it is killed, so a hung build
does not hold a runner slot indefinitely. Accepts a Go duration
string (`30m`, `1h`, `2h30m`). Enforced by the runner process, so
it only fires while that process is healthy.
'';
};
};
config = lib.mkIf cfg.enable {
assertions = [
{
assertion = haveClientIdentity;
message = ''
services.hyperhive.deploy.ci-runner.enable requires the runner's
swarm-secret-store client identity: set both
services.hyperhive.deploy.bao.ciRunnerClientCertFile
services.hyperhive.deploy.bao.ciRunnerClientKeyFile
swarm-bao-ci-runner.service reads the forge's runner registration
token with it, and the runner cannot register without that token.
On the host that runs the store, glue-bao-tls.nix supplies both as
defaults. Elsewhere, copy ci-runner.pem and ci-runner-key.pem from
the store host's /var/lib/swarm-bao-pki/ and name them here.
'';
}
];
# The container hangs off the bridge, takes its address from the
# gateway's DHCP and resolves through its dnsmasq. The gateway also runs
# `hive-tls-ca`, which writes the trust bundle bound in below.
services.hyperhive.network.enable = lib.mkDefault true;
services.hyperhive.gateway.enable = lib.mkDefault true;
services.hyperhive.gateway.dns.enable = lib.mkDefault true;
# Shaped after ./swarm-grafana.nix's `swarm-bao-grafana-oidc`, except that
# a missing token fails too: swarm-controller publishes it at its start and
# every five minutes, so a retry is what turns "no value there" into one.
systemd.services.swarm-bao-ci-runner = {
description = "fetch the CI runner's registration token from the swarm secret store";
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
before = [ "container@${machine}.service" ];
wantedBy = [
"multi-user.target"
"container@${machine}.service"
];
path = [
baoDeploy.package
pkgs.coreutils
pkgs.systemd
];
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = "swarm-bao-ci-runner";
TimeoutStartSec = 30;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.ciRunnerClientCertFile;
BAO_CLIENT_KEY = baoDeploy.ciRunnerClientKeyFile;
}
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
${atomicWriteSecret}
${refreshConsumer}
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# `-token-only` answers on stdout and needs no token helper.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with the CI runner's certificate; leaving its registration token as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! token="$(bao kv get -field=value ${lib.escapeShellArg storePath} 2>"$err")"; then
echo "swarm-bao did not return ${storePath}; retrying until swarm-controller publishes it." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
if [ -z "$token" ]; then
echo "swarm-bao returned an empty ${storePath}; retrying." >&2
exit 1
fi
# Root-owned: systemd reads an `EnvironmentFile` as root, before the
# runner drops to its dynamic user.
install -d -m 0700 ${lib.escapeShellArg (builtins.dirOf hostTokenFile)}
if secret_differs ${lib.escapeShellArg hostTokenFile} "TOKEN=$token"; then
atomic_write_secret 0600 "" ${lib.escapeShellArg hostTokenFile} "TOKEN=$token"
fi
# `EnvironmentFile=` is read once, at start. A changed token makes the
# upstream pre-start step register again.
refresh_consumer ${lib.escapeShellArg machine} ${runnerUnit}.service ${lib.escapeShellArg hostTokenFile}
'';
};
systemd.services."container@${machine}" = caTrust.containerOrdering;
containers.${machine} = {
autoStart = true;
ephemeral = false;
# Journal files on the host, not inside the container: nixpkgs hardcodes
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it.
extraFlags = [ "--link-journal=host" ];
inherit privateNetwork;
hostBridge = networkCfg.bridgeName;
bindMounts = {
# Builds go through the HOST nix-daemon, so they inherit its
# buildMachines and max-jobs. The directory rather than the socket
# file, so the mount survives a daemon restart replacing the inode.
"/nix/var/nix/daemon-socket" = {
hostPath = "/nix/var/nix/daemon-socket";
isReadOnly = false;
};
}
// caTrust.bindMount;
config =
{ pkgs, lib, ... }:
{
imports = [
../container-modules/swarm-container.nix
caBundleModule
];
# Its own netns, so it keeps its own firewall; resolvconf stays on
# and serves `networking.nameservers` below.
services.hyperhive.swarmContainer = {
inherit privateNetwork;
writesOwnResolvConf = false;
};
# The gateway's dnsmasq on the bridge answers the forge's name with
# the bridge IP where this host runs the forge, and forwards it
# upstream everywhere else.
networking.nameservers = [ networkCfg.bridgeIp ];
networking.interfaces.eth0.useDHCP = true;
# nspawn containers can't create user namespaces, so nix sandboxing
# fails; see docs/process/gotchas.md. `fallback` is a client-side
# option sent per daemon connection, so the host's value does not
# reach builds started here.
nix.settings.sandbox-fallback = lib.mkForce true;
nix.settings.fallback = true;
nix.settings.experimental-features = [
"nix-command"
"flakes"
];
# Both the service and its socket are off: the socket unit owns
# `/nix/var/nix/daemon-socket/socket` and would race the bind mount.
systemd.globalEnvironment.NIX_REMOTE = "daemon";
systemd.services.nix-daemon.enable = false;
systemd.sockets.nix-daemon.enable = false;
systemd.tmpfiles.rules = [
"d ${stateDir} 0700 root root -"
# `EnvironmentFile=` fails the unit when the file is missing; an
# empty one leaves `TOKEN` unset for the precond to report.
"f ${tokenFile} 0600 root root -"
];
services.gitea-actions-runner.package = cfg.package;
services.gitea-actions-runner.instances.${instance} = {
enable = true;
inherit (cfg) name labels;
url = forgeUrl;
inherit tokenFile;
settings = {
runner.capacity = cfg.concurrency;
runner.timeout = cfg.jobTimeout;
};
};
environment.systemPackages = [
pkgs.git
pkgs.nix
];
systemd.services.${runnerUnit} = {
# `path` adds to the PATH the upstream module builds instead of
# replacing it.
path = [ pkgs.nix ];
# The upstream unit is `DynamicUser`, whose state dir systemd mounts
# `noexec`, so a job could not execute what it builds in its
# workspace. `ExecPaths=` resolves against the host root, where
# `/var/lib/gitea-runner` is a symlink into `private/`; `-` makes
# whichever spelling is absent a no-op.
serviceConfig.ExecPaths = [
"-/var/lib/gitea-runner"
"-/var/lib/private/gitea-runner"
];
environment = lib.mkIf caTrust.useSelfSigned {
NODE_EXTRA_CA_CERTS = caTrust.caContainerPath;
};
# Runs before upstream's pre-start step, which registers whenever
# `.runner` is absent or the token or labels changed. This one
# removes `.runner` in the two cases upstream cannot see, and holds
# the unit down until there is a token.
#
# `+`: root, because `.runner`, the token and the system journal
# are all out of the dynamic user's reach.
serviceConfig.ExecStartPre = lib.mkMerge [
(lib.mkBefore [
"+${pkgs.writeShellScript "swarm-ci-runner-precond" ''
set -eu
runner=${runnerFile}
started=${daemonStartedFile}
# The forge refused the registration the daemon last started
# with. Read on the next start rather than when the daemon
# exits, so journald has its last lines by then.
if [ -f "$started" ]; then
since="$(< "$started")"
${pkgs.coreutils}/bin/rm -f "$started"
if ${pkgs.systemd}/bin/journalctl -u ${runnerUnit}.service --since "@$since" -o cat --no-pager \
| ${pkgs.gnugrep}/bin/grep -qF ${lib.escapeShellArg rejectedMessage}; then
echo "the forge has no record of this runner's registration; registering again" >&2
${pkgs.coreutils}/bin/rm -f "$runner"
fi
fi
# `.runner` records the address it registered at, and the
# daemon dials that one, not the configured one.
if [ -f "$runner" ] \
&& [ "$(${pkgs.jq}/bin/jq -r .address "$runner")" != ${lib.escapeShellArg forgeUrl} ]; then
echo "$runner names another forge address; registering again at ${forgeUrl}" >&2
${pkgs.coreutils}/bin/rm -f "$runner"
fi
# Upstream registers on a changed token hash, an empty token
# included, and a registration with no token fails.
if [ -z "''${TOKEN:-}" ]; then
echo "no registration token yet; swarm-bao-ci-runner on the host delivers it" >&2
exit 1
fi
''}"
])
(lib.mkAfter [
"+${pkgs.writeShellScript "swarm-ci-runner-mark-start" ''
${pkgs.coreutils}/bin/date +%s > ${daemonStartedFile}
''}"
])
];
# Restarted until the token arrives, however long the store and
# the controller take. `mkForce`: upstream sets both.
serviceConfig.Restart = lib.mkForce "on-failure";
serviceConfig.RestartSec = lib.mkForce 15;
startLimitIntervalSec = 0;
};
};
};
};
}

1
nix/host-modules/swarm.nix
View file

@ -54,6 +54,7 @@ let
deployCfg.bao.natsCommonName
deployCfg.bao.granterCommonName
deployCfg.bao.natsAuthCommonName
deployCfg.bao.ciRunnerCommonName
]
# The per-hive subjects, spelled out per hive rather than as the prefix.
# The prefix alone would reserve the wrong string: the role for hive