vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-ci-runner: load the registration token in a register unit of its own

Browse source 
systemd hands a unit's credentials to every process of that unit, and a
host job is a process of the runner daemon, so the `LoadCredential=` on
gitea-runner-swarm still let any workflow read the registration token
from `$CREDENTIALS_DIRECTORY`.

swarm-ci-runner-register.service, a oneshot under the daemon's dynamic
user and state dir, now holds the only `LoadCredential=` and runs the
precond and `act_runner register` (only when `.runner` is missing). The
daemon unit has no credential, no `EnvironmentFile` and no `TOKEN`; its
pre-start step starts the register unit, so a missing token fails the
daemon's start and `Restart=on-failure` retries it.

The precond compares `.runner`'s labels in the form the daemon rewrites
them to (`nix` -> `nix:host`); compared verbatim, a scheme-less label
re-registered the runner on every start.

tmpfiles removes the unused `registration-token.env` and upstream's
`.token-hash`/`.labels` markers.
This commit is contained in:
atlas 2026-10-05 16:33:24 +02:00
commit 8ffceea8e3
4 changed files with 184 additions and 100 deletions
Download patch file Download diff file Expand all files Collapse all files

30
docs/scheduler/ci.md
View file

@ -130,7 +130,7 @@ can't hold the runner's single slot indefinitely).
- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI. - **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI.
- **Non-ephemeral**: runner credentials (`.runner`) persist across restarts, written to the container's state dir on registration and reused after. - **Non-ephemeral**: runner credentials (`.runner`) persist across restarts, written to the container's state dir on registration and reused after.
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/process/gotchas.md`. - **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/process/gotchas.md`.
- **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path. Only the register step reads the registration token; the daemon and the jobs it runs never have it in their environment. - **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path. Only `swarm-ci-runner-register.service` loads the registration token; the daemon unit and the jobs it runs have it neither in their environment nor among their credentials.
## Registration ## Registration
@ -146,13 +146,18 @@ can't hold the runner's single slot indefinitely).
writes the token to a root-only file inside the container, and restarts writes the token to a root-only file inside the container, and restarts
the runner when the value changes. It runs at boot and on deploy, not on a the runner when the value changes. It runs at boot and on deploy, not on a
timer. timer.
3. **The runner unit** (`gitea-runner-swarm` in the container) loads that 3. **`swarm-ci-runner-register.service`** in the container is the only unit
file with `LoadCredential=`, not as an environment variable: for a `host` that loads that file, with `LoadCredential=`. systemd hands a unit's
label, act_runner hands its whole environment to every job step. A credentials to all of its processes, and for a `host` label act_runner
register step before the daemon reads the token from runs every job step as a process of the daemon with the daemon's whole
`$CREDENTIALS_DIRECTORY` and runs `act_runner register` only when `.runner` environment, so the daemon unit carries the token in neither. The
is missing; with a `.runner` in place the daemon starts without the token. register unit is a oneshot under the daemon's dynamic user and state
A pre-start step ahead of it removes `.runner` in three cases: directory. It reads the token from `$CREDENTIALS_DIRECTORY` and runs
`act_runner register` only when `.runner` is missing, so the `.runner` it
writes is the daemon's own. **The runner unit** (`gitea-runner-swarm`)
starts it in a pre-start step and runs the daemon once it succeeds.
A root step of the register unit, ahead of registering, removes `.runner`
in three cases:
- The forge has no record of the runner. Forgejo answers every call from an - The forge has no record of the runner. Forgejo answers every call from an
unknown runner with `unregistered runner`, and the daemon exits 1 when its unknown runner with `unregistered runner`, and the daemon exits 1 when its
startup `Declare` gets that answer. On the next start the step finds the startup `Declare` gets that answer. On the next start the step finds the
@ -161,11 +166,14 @@ can't hold the runner's single slot indefinitely).
and logging the error without exiting; the next restart of the unit or and logging the error without exiting; the next restart of the unit or
container registers it again. container registers it again.
- `.runner` names another forge address than the configured one. - `.runner` names another forge address than the configured one.
- `.runner` names other labels than the configured ones. - `.runner` names other labels than the configured ones, compared in the
form the daemon writes back: `<name>:<scheme>`, with `host` for a
label that names no scheme.
A changed token alone doesn't register again: the daemon authenticates with A changed token alone doesn't register again: the daemon authenticates with
the per-runner token in `.runner`. Without `.runner` and with no token yet the per-runner token in `.runner`. Without `.runner` and with no token yet
the register step fails, and `Restart=on-failure` retries every 15 seconds the register unit fails and the runner unit's pre-start step with it;
`Restart=on-failure` on the runner unit starts both again every 15 seconds,
with no start limit. with no start limit.
Host jobs run as the runner's own user, so they can still read `.runner` Host jobs run as the runner's own user, so they can still read `.runner`
@ -231,7 +239,7 @@ A malicious `default.nix` or build script in a PR can therefore:
- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at `https://<forge.domain>` (no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic). - **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at `https://<forge.domain>` (no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic).
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials. - **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which only the runner's register step reads, so a job step doesn't inherit it. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls. No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which only `swarm-ci-runner-register.service` loads, so a job step can't read it. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls.
Note: `nix flake check --no-build` (eval-only) reduces the attack surface but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path. Note: `nix flake check --no-build` (eval-only) reduces the attack surface but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.

32
docs/swarm/credentials.md
View file

@ -55,22 +55,22 @@ of the cell says how.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | | store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | | `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts | | `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs | | `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated | | `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only the runner's register step loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ❌ `swarm-bao-ci-runner` reads it when it starts, at boot or deploy, and restarts the runner when the value changed | | `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ❌ `swarm-bao-ci-runner` reads it when it starts, at boot or deploy, and restarts the runner when the value changed |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated | | _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->

168
nix/host-modules/swarm-ci-runner.nix
View file

@ -36,6 +36,8 @@ let
# The upstream module names its unit `gitea-runner-<instance>`. # The upstream module names its unit `gitea-runner-<instance>`.
instance = "swarm"; instance = "swarm";
runnerUnit = "gitea-runner-${instance}"; runnerUnit = "gitea-runner-${instance}";
# Holds the registration token and registers; see its definition.
registerUnit = "swarm-ci-runner-register";
# What the runner registers against and records in `.runner`. The precond # What the runner registers against and records in `.runner`. The precond
# below compares the two, so both come from here. # below compares the two, so both come from here.
@ -49,7 +51,10 @@ let
caBundleModule = caTrust.trustBundle { caBundleModule = caTrust.trustBundle {
inherit pkgs; inherit pkgs;
name = machine; name = machine;
consumers = [ runnerUnit ]; consumers = [
runnerUnit
registerUnit
];
}; };
# The upstream unit is `DynamicUser`, so its state dir is # The upstream unit is `DynamicUser`, so its state dir is
@ -58,9 +63,8 @@ let
runnerFile = "/var/lib/gitea-runner/${instance}/.runner"; runnerFile = "/var/lib/gitea-runner/${instance}/.runner";
# Inside the container, root-only. The token file holds the bare token and # Inside the container, root-only. The token file holds the bare token and
# reaches the runner unit only as a `LoadCredential=`, which only the # is a `LoadCredential=` of the register unit alone; the host unit below
# register step reads; the host unit below writes it through the # writes it through the container's root, at `hostTokenFile`.
# container's root, at `hostTokenFile`.
stateDir = "/var/lib/swarm-ci-runner"; stateDir = "/var/lib/swarm-ci-runner";
tokenFile = "${stateDir}/registration-token"; tokenFile = "${stateDir}/registration-token";
tokenCredential = "registration-token"; tokenCredential = "registration-token";
@ -295,8 +299,20 @@ in
instanceCfg = runnerCfg.instances.${instance}; instanceCfg = runnerCfg.instances.${instance};
# The same file upstream generates and hands the daemon. # The same file upstream generates and hands the daemon.
runnerConfigFile = (pkgs.formats.yaml { }).generate "config.yaml" instanceCfg.settings; runnerConfigFile = (pkgs.formats.yaml { }).generate "config.yaml" instanceCfg.settings;
# In the form the precond reads back from `.runner`. # In the form the daemon rewrites `.runner`'s labels to on start
registeredLabels = lib.concatStringsSep "," (lib.sort lib.lessThan instanceCfg.labels); # (act_runner's `labels.Parse` then `ToStrings`): `<name>:<scheme>`
# with `host` for a missing scheme, then `:<arg>` if there is one.
normaliseLabel =
label:
let
parts = lib.splitString ":" label;
scheme = if lib.length parts >= 2 then lib.elemAt parts 1 else "host";
arg = lib.concatStringsSep ":" (lib.drop 2 parts);
in
"${lib.head parts}:${scheme}" + lib.optionalString (arg != "") ":${arg}";
registeredLabels = lib.concatStringsSep "," (
lib.sort lib.lessThan (map normaliseLabel instanceCfg.labels)
);
in in
{ {
imports = [ imports = [
@ -337,9 +353,13 @@ in
systemd.tmpfiles.rules = [ systemd.tmpfiles.rules = [
"d ${stateDir} 0700 root root -" "d ${stateDir} 0700 root root -"
# `LoadCredential=` fails the unit when the file is missing; an # `LoadCredential=` fails the unit when the file is missing; an
# empty one lets a registered runner start before any token has # empty one lets the register unit report the missing token.
# arrived, and the register step report the missing token.
"f ${tokenFile} 0600 root root -" "f ${tokenFile} 0600 root root -"
# Unused: nothing in this module writes or reads them. The first
# holds a token; the others are upstream's re-registration markers.
"r ${stateDir}/registration-token.env"
"r /var/lib/private/gitea-runner/${instance}/.token-hash"
"r /var/lib/private/gitea-runner/${instance}/.labels"
]; ];
services.gitea-actions-runner.package = cfg.package; services.gitea-actions-runner.package = cfg.package;
@ -349,7 +369,7 @@ in
inherit (cfg) name labels; inherit (cfg) name labels;
url = forgeUrl; url = forgeUrl;
# Upstream refuses an instance without a token source, and turns # Upstream refuses an instance without a token source, and turns
# this one into the `EnvironmentFile=` the unit below drops. # this one into the `EnvironmentFile=` the daemon unit drops.
inherit tokenFile; inherit tokenFile;
settings = { settings = {
runner.capacity = cfg.concurrency; runner.capacity = cfg.concurrency;
@ -362,36 +382,37 @@ in
pkgs.nix pkgs.nix
]; ];
systemd.services.${runnerUnit} = { # The only unit that can read the registration token: systemd hands
# `path` adds to the PATH the upstream module builds instead of # a credential to every process of the unit that loads it, and the
# replacing it. # daemon's processes include every host job. Leaves a usable
path = [ pkgs.nix ]; # `.runner` behind or fails.
# The upstream unit is `DynamicUser`, whose state dir systemd mounts #
# `noexec`, so a job could not execute what it builds in its # The daemon's dynamic user and state dir: units naming the same
# workspace. `ExecPaths=` resolves against the host root, where # `DynamicUser` share its uid, so `.runner` is the daemon's own.
# `/var/lib/gitea-runner` is a symlink into `private/`; `-` makes systemd.services.${registerUnit} = {
# whichever spelling is absent a no-op. description = "register the CI runner with the forge when it has no .runner";
serviceConfig.ExecPaths = [ wants = [ "network-online.target" ];
"-/var/lib/gitea-runner" after = [ "network-online.target" ];
"-/var/lib/private/gitea-runner" environment = {
]; HOME = "/var/lib/gitea-runner/${instance}";
environment = lib.mkIf caTrust.useSelfSigned { # In the form the daemon rewrites `.runner`'s labels to.
NODE_EXTRA_CA_CERTS = caTrust.caContainerPath; SWARM_CI_RUNNER_LABELS = registeredLabels;
}; };
# The registration token reaches the unit only as a credential, serviceConfig = {
# which the register step reads. A host job inherits the daemon's Type = "oneshot";
# whole environment, so neither carries it. DynamicUser = true;
serviceConfig.EnvironmentFile = lib.mkForce [ ]; User = "gitea-runner";
serviceConfig.LoadCredential = [ "${tokenCredential}:${tokenFile}" ]; StateDirectory = "gitea-runner";
# One registration path: upstream's pre-start step registers from LoadCredential = [ "${tokenCredential}:${tokenFile}" ];
# `$TOKEN`, so this list replaces it. # `register` retries an unreachable forge without end; under the
serviceConfig.ExecStartPre = lib.mkForce [ # daemon's own start timeout, so the failure is this unit's.
TimeoutStartSec = 60;
# Removes `.runner` in the cases where reusing it would fail, so # Removes `.runner` in the cases where reusing it would fail, so
# the register step after it registers again. # the register step registers again.
# #
# `+`: root, because the system journal is out of the dynamic # `+`: root, because the system journal is out of the dynamic
# user's reach. # user's reach.
"+${pkgs.writeShellScript "swarm-ci-runner-precond" '' ExecStartPre = "+${pkgs.writeShellScript "swarm-ci-runner-precond" ''
set -eu set -eu
runner=${runnerFile} runner=${runnerFile}
started=${daemonStartedFile} started=${daemonStartedFile}
@ -420,35 +441,62 @@ in
# The daemon declares the labels `.runner` records, not the # The daemon declares the labels `.runner` records, not the
# configured ones. # configured ones.
if [ -f "$runner" ] \ if [ -f "$runner" ] \
&& [ "$(${pkgs.jq}/bin/jq -r '.labels | sort | join(",")' "$runner")" != ${lib.escapeShellArg registeredLabels} ]; then && [ "$(${pkgs.jq}/bin/jq -r '.labels | sort | join(",")' "$runner")" != "$SWARM_CI_RUNNER_LABELS" ]; then
echo "$runner names other labels; registering again with ${registeredLabels}" >&2 echo "$runner names other labels; registering again with $SWARM_CI_RUNNER_LABELS" >&2
${pkgs.coreutils}/bin/rm -f "$runner" ${pkgs.coreutils}/bin/rm -f "$runner"
fi fi
''}" ''}";
# Registers only when `.runner` is absent. As the dynamic user, };
# so `.runner` is the daemon's own. The derivation itself rather script = ''
# than its path, so ../module-eval reads its text without building. set -u
(pkgs.writeShellScript "swarm-ci-runner-register" '' dir="$STATE_DIRECTORY/${instance}"
set -eu mkdir -p "$dir"
dir="$STATE_DIRECTORY/${instance}" cd "$dir"
${pkgs.coreutils}/bin/mkdir -p "$dir" if [ -e .runner ]; then
cd "$dir" exit 0
if [ -e .runner ]; then fi
exit 0
fi
token="$(< "$CREDENTIALS_DIRECTORY/${tokenCredential}")" token="$(< "$CREDENTIALS_DIRECTORY/${tokenCredential}")"
if [ -z "$token" ]; then if [ -z "$token" ]; then
echo "no registration token yet; swarm-bao-ci-runner on the host delivers it" >&2 echo "no registration token yet; swarm-bao-ci-runner on the host delivers it" >&2
exit 1 exit 1
fi fi
exec ${lib.getExe runnerCfg.package} register --no-interactive \ exec ${lib.getExe runnerCfg.package} register --no-interactive \
--instance ${lib.escapeShellArg instanceCfg.url} \ --instance ${lib.escapeShellArg instanceCfg.url} \
--token "$token" \ --token "$token" \
--name ${lib.escapeShellArg instanceCfg.name} \ --name ${lib.escapeShellArg instanceCfg.name} \
--labels ${lib.escapeShellArg (lib.concatStringsSep "," instanceCfg.labels)} \ --labels ${lib.escapeShellArg (lib.concatStringsSep "," instanceCfg.labels)} \
--config ${runnerConfigFile} --config ${runnerConfigFile}
'') '';
};
systemd.services.${runnerUnit} = {
# `path` adds to the PATH the upstream module builds instead of
# replacing it.
path = [ pkgs.nix ];
# The upstream unit is `DynamicUser`, whose state dir systemd mounts
# `noexec`, so a job could not execute what it builds in its
# workspace. `ExecPaths=` resolves against the host root, where
# `/var/lib/gitea-runner` is a symlink into `private/`; `-` makes
# whichever spelling is absent a no-op.
serviceConfig.ExecPaths = [
"-/var/lib/gitea-runner"
"-/var/lib/private/gitea-runner"
];
environment = lib.mkIf caTrust.useSelfSigned {
NODE_EXTRA_CA_CERTS = caTrust.caContainerPath;
};
# A host job inherits the daemon's whole environment and can read
# its credentials, so the daemon carries the token in neither.
serviceConfig.EnvironmentFile = lib.mkForce [ ];
# Replaces upstream's pre-start step, which registers from
# `$TOKEN`, so the register unit is the one registration path.
#
# Started from here rather than through `Requires=`: a failed
# requirement fails the start job, which `Restart=` does not retry,
# while a failed pre-start step does.
serviceConfig.ExecStartPre = lib.mkForce [
"+${pkgs.systemd}/bin/systemctl start ${registerUnit}.service"
"+${pkgs.writeShellScript "swarm-ci-runner-mark-start" '' "+${pkgs.writeShellScript "swarm-ci-runner-mark-start" ''
${pkgs.coreutils}/bin/date +%s > ${daemonStartedFile} ${pkgs.coreutils}/bin/date +%s > ${daemonStartedFile}
''}" ''}"

54
nix/module-eval/forge-placement.nix
View file

@ -47,10 +47,14 @@ let
}; };
# The runner alone: no hive, no forge, no store, its store identity placed # The runner alone: no hive, no forge, no store, its store identity placed
# by hand. # by hand. `nix` is a label without a scheme.
ciAlone = hive { ciAlone = hive {
deploy.hive-controller.enable = false; deploy.hive-controller.enable = false;
deploy.ci-runner.enable = true; deploy.ci-runner.enable = true;
deploy.ci-runner.labels = [
"hive-ci:host"
"nix"
];
deploy.bao.ciRunnerClientCertFile = "/etc/pki/bao-ci-runner.pem"; deploy.bao.ciRunnerClientCertFile = "/etc/pki/bao-ci-runner.pem";
deploy.bao.ciRunnerClientKeyFile = "/etc/pki/bao-ci-runner-key.pem"; deploy.bao.ciRunnerClientKeyFile = "/etc/pki/bao-ci-runner-key.pem";
}; };
@ -82,7 +86,8 @@ let
forgeSettings = forgeHere.containers.hive-forge.config.services.forgejo.settings; forgeSettings = forgeHere.containers.hive-forge.config.services.forgejo.settings;
runnerService = ciAlone.containers.swarm-ci-runner.config.systemd.services.gitea-runner-swarm; runnerService = ciAlone.containers.swarm-ci-runner.config.systemd.services.gitea-runner-swarm;
registerStep = lib.findFirst lib.isDerivation null runnerService.serviceConfig.ExecStartPre; registerService =
ciAlone.containers.swarm-ci-runner.config.systemd.services.swarm-ci-runner-register;
cases = [ cases = [
{ {
@ -154,9 +159,10 @@ let
ok = refusedOver ciNoIdentity "deploy.bao.ciRunnerClientCertFile"; ok = refusedOver ciNoIdentity "deploy.bao.ciRunnerClientCertFile";
} }
{ {
# A host job inherits the daemon's whole environment, and a token there # systemd hands a credential to every process of the unit that loads
# lets any workflow register runners against the forge. # it, and a host job is one of the daemon's processes and inherits its
name = "the CI runner daemon carries no registration token in its environment"; # environment; a token in either lets any workflow register runners.
name = "the CI runner daemon can read no registration token";
ok = ok =
let let
sc = runnerService.serviceConfig; sc = runnerService.serviceConfig;
@ -164,26 +170,48 @@ let
!(runnerService.environment ? TOKEN) !(runnerService.environment ? TOKEN)
&& lib.toList (sc.EnvironmentFile or [ ]) == [ ] && lib.toList (sc.EnvironmentFile or [ ]) == [ ]
&& !(lib.any (lib.hasPrefix "TOKEN=") (lib.toList (sc.Environment or [ ]))) && !(lib.any (lib.hasPrefix "TOKEN=") (lib.toList (sc.Environment or [ ])))
&& sc.LoadCredential == [ "registration-token:/var/lib/swarm-ci-runner/registration-token" ]; && !(sc ? LoadCredential)
&& !(sc ? LoadCredentialEncrypted)
&& !(sc ? SetCredential);
} }
{ {
# The only reader of the credential, and the only registration path. name = "the CI register unit alone loads the registration token, and runs before the daemon";
name = "the CI runner registers from its credential, and only without a .runner";
ok = ok =
let let
text = registerStep.text; sc = registerService.serviceConfig;
beforeRegister = lib.head (lib.splitString "register --no-interactive" text);
in in
registerStep != null sc.LoadCredential == [ "registration-token:/var/lib/swarm-ci-runner/registration-token" ]
&& sc.Type == "oneshot"
&& sc.User == runnerService.serviceConfig.User
&& sc.DynamicUser
&& sc.StateDirectory == runnerService.serviceConfig.StateDirectory
&& lib.any (
s: lib.hasSuffix "/bin/systemctl start swarm-ci-runner-register.service" s
) runnerService.serviceConfig.ExecStartPre
&& !(lib.any ( && !(lib.any (
s: lib.hasInfix "gitea-register-runner" (toString s) s: lib.hasInfix "gitea-register-runner" (toString s)
) runnerService.serviceConfig.ExecStartPre) ) runnerService.serviceConfig.ExecStartPre);
&& lib.hasInfix "if [ -e .runner ]; then\n exit 0\nfi" beforeRegister }
{
name = "the CI register unit registers from its credential, and only without a .runner";
ok =
let
text = registerService.script;
beforeRegister = lib.head (lib.splitString "register --no-interactive" text);
in
lib.hasInfix "if [ -e .runner ]; then\n exit 0\nfi" beforeRegister
&& lib.hasInfix ''token="$(< "$CREDENTIALS_DIRECTORY/registration-token")"'' beforeRegister && lib.hasInfix ''token="$(< "$CREDENTIALS_DIRECTORY/registration-token")"'' beforeRegister
&& lib.hasInfix ''--token "$token"'' text && lib.hasInfix ''--token "$token"'' text
&& !(lib.hasInfix "export" text) && !(lib.hasInfix "export" text)
&& !(lib.hasInfix "TOKEN" text); && !(lib.hasInfix "TOKEN" text);
} }
{
# The daemon rewrites `.runner`'s labels to this form on start; compared
# against the configured strings verbatim, a scheme-less label would
# discard `.runner` and register a new runner on every start.
name = "the CI runner compares .runner's labels in the form the daemon writes them";
ok = registerService.environment.SWARM_CI_RUNNER_LABELS == "hive-ci:host,nix:host";
}
{ {
# A removed option evaluates to an assertion naming the replacement. # A removed option evaluates to an assertion naming the replacement.
name = "a CI option under deploy.forgejo names its replacement under deploy.ci-runner"; name = "a CI option under deploy.forgejo names its replacement under deploy.ci-runner";