vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-ci-runner: give the registration token only to the register step

Browse source 
The runner got its registration token through an `EnvironmentFile`
(`TOKEN=…`), so the daemon carried it for its whole life, and for a
`host` label act_runner copies its environment into every job step:
any workflow could read a token that registers runners on the forge.

The host unit now writes the bare token, and the runner unit loads it
with `LoadCredential=`. The upstream pre-start step, which registered
from `$TOKEN`, is replaced by one register step that reads the token
from `$CREDENTIALS_DIRECTORY` and runs `act_runner register` only when
`.runner` is missing, with the instance URL, name, labels and config
file upstream used. The precond now also removes `.runner` when its
labels differ from the configured ones, which upstream's step covered.
A changed token alone no longer re-registers.

Host jobs can still read `.runner` itself (#4979).

Closes #4974
This commit is contained in:
atlas 2026-10-05 16:06:19 +02:00
commit 0302b29ca4
4 changed files with 158 additions and 71 deletions
Download patch file Download diff file Expand all files Collapse all files

31
docs/scheduler/ci.md
View file

@ -130,7 +130,7 @@ can't hold the runner's single slot indefinitely).
- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI. - **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI.
- **Non-ephemeral**: runner credentials (`.runner`) persist across restarts, written to the container's state dir on registration and reused after. - **Non-ephemeral**: runner credentials (`.runner`) persist across restarts, written to the container's state dir on registration and reused after.
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/process/gotchas.md`. - **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/process/gotchas.md`.
- **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path. - **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path. Only the register step reads the registration token; the daemon and the jobs it runs never have it in their environment.
## Registration ## Registration
@ -143,13 +143,16 @@ can't hold the runner's single slot indefinitely).
2. **`swarm-bao-ci-runner.service`** on the runner's host logs in to the store 2. **`swarm-bao-ci-runner.service`** on the runner's host logs in to the store
with the `ci-runner` leaf, whose `swarm-ci-runner` role reads that one path with the `ci-runner` leaf, whose `swarm-ci-runner` role reads that one path
and nothing else, retrying every 30 seconds until the token is there. It and nothing else, retrying every 30 seconds until the token is there. It
writes `TOKEN=<value>` to the runner's `EnvironmentFile` inside the writes the token to a root-only file inside the container, and restarts
container, and restarts the runner when the value changes. It runs at boot the runner when the value changes. It runs at boot and on deploy, not on a
and on deploy, not on a timer. timer.
3. **The runner unit** (`gitea-runner-swarm` in the container) registers in 3. **The runner unit** (`gitea-runner-swarm` in the container) loads that
the upstream pre-start step whenever `.runner` is missing or the token or file with `LoadCredential=`, not as an environment variable: for a `host`
labels changed. A pre-start step of this module's, ahead of it, removes label, act_runner hands its whole environment to every job step. A
`.runner` in two cases upstream doesn't check: register step before the daemon reads the token from
`$CREDENTIALS_DIRECTORY` and runs `act_runner register` only when `.runner`
is missing; with a `.runner` in place the daemon starts without the token.
A pre-start step ahead of it removes `.runner` in three cases:
- The forge has no record of the runner. Forgejo answers every call from an - The forge has no record of the runner. Forgejo answers every call from an
unknown runner with `unregistered runner`, and the daemon exits 1 when its unknown runner with `unregistered runner`, and the daemon exits 1 when its
startup `Declare` gets that answer. On the next start the step finds the startup `Declare` gets that answer. On the next start the step finds the
@ -158,9 +161,15 @@ can't hold the runner's single slot indefinitely).
and logging the error without exiting; the next restart of the unit or and logging the error without exiting; the next restart of the unit or
container registers it again. container registers it again.
- `.runner` names another forge address than the configured one. - `.runner` names another forge address than the configured one.
- `.runner` names other labels than the configured ones.
Until a token arrives the step fails, and `Restart=on-failure` retries every A changed token alone doesn't register again: the daemon authenticates with
15 seconds with no start limit. the per-runner token in `.runner`. Without `.runner` and with no token yet
the register step fails, and `Restart=on-failure` retries every 15 seconds
with no start limit.
Host jobs run as the runner's own user, so they can still read `.runner`
and act as this runner; see the security section below.
A new registration is a new runner on the forge: the previous entry stays in A new registration is a new runner on the forge: the previous entry stays in
the forge admin panel as offline until an admin deletes it. the forge admin panel as offline until an admin deletes it.
@ -222,7 +231,7 @@ A malicious `default.nix` or build script in a PR can therefore:
- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at `https://<forge.domain>` (no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic). - **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at `https://<forge.domain>` (no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic).
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials. - **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which reaches the container's env-file. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls. No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which only the runner's register step reads, so a job step doesn't inherit it. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls.
Note: `nix flake check --no-build` (eval-only) reduces the attack surface but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path. Note: `nix flake check --no-build` (eval-only) reduces the attack surface but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.

2
docs/swarm/credentials.md
View file

@ -69,7 +69,7 @@ of the cell says how.
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs | | `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated | | `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into the runner's `EnvironmentFile`; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ❌ `swarm-bao-ci-runner` reads it when it starts, at boot or deploy, and restarts the runner when the value changed | | `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only the runner's register step loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ❌ `swarm-bao-ci-runner` reads it when it starts, at boot or deploy, and restarts the runner when the value changed |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated | | _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->

158
nix/host-modules/swarm-ci-runner.nix
View file

@ -57,11 +57,13 @@ let
# it, which is the spelling a step run as root (`+`) can follow. # it, which is the spelling a step run as root (`+`) can follow.
runnerFile = "/var/lib/gitea-runner/${instance}/.runner"; runnerFile = "/var/lib/gitea-runner/${instance}/.runner";
# Inside the container, root-only. The token file is the runner's # Inside the container, root-only. The token file holds the bare token and
# `EnvironmentFile` (`TOKEN=<value>`); the host unit below writes it through # reaches the runner unit only as a `LoadCredential=`, which only the
# the container's root, at `hostTokenFile`. # register step reads; the host unit below writes it through the
# container's root, at `hostTokenFile`.
stateDir = "/var/lib/swarm-ci-runner"; stateDir = "/var/lib/swarm-ci-runner";
tokenFile = "${stateDir}/registration-token.env"; tokenFile = "${stateDir}/registration-token";
tokenCredential = "registration-token";
hostTokenFile = "/var/lib/nixos-containers/${machine}${tokenFile}"; hostTokenFile = "/var/lib/nixos-containers/${machine}${tokenFile}";
# The epoch second the daemon last started. The precond reads the daemon's # The epoch second the daemon last started. The precond reads the daemon's
# journal from there. # journal from there.
@ -246,15 +248,15 @@ in
exit 1 exit 1
fi fi
# Root-owned: systemd reads an `EnvironmentFile` as root, before the # Root-owned: systemd reads a `LoadCredential=` source as root, before
# runner drops to its dynamic user. # the runner drops to its dynamic user.
install -d -m 0700 ${lib.escapeShellArg (builtins.dirOf hostTokenFile)} install -d -m 0700 ${lib.escapeShellArg (builtins.dirOf hostTokenFile)}
if secret_differs ${lib.escapeShellArg hostTokenFile} "TOKEN=$token"; then if secret_differs ${lib.escapeShellArg hostTokenFile} "$token"; then
atomic_write_secret 0600 "" ${lib.escapeShellArg hostTokenFile} "TOKEN=$token" atomic_write_secret 0600 "" ${lib.escapeShellArg hostTokenFile} "$token"
fi fi
# `EnvironmentFile=` is read once, at start. A changed token makes the # `LoadCredential=` copies the file at start, so a runner still waiting
# upstream pre-start step register again. # to register sees a new token only once it restarts.
refresh_consumer ${lib.escapeShellArg machine} ${runnerUnit}.service ${lib.escapeShellArg hostTokenFile} refresh_consumer ${lib.escapeShellArg machine} ${runnerUnit}.service ${lib.escapeShellArg hostTokenFile}
''; '';
}; };
@ -282,7 +284,20 @@ in
// caTrust.bindMount; // caTrust.bindMount;
config = config =
{ pkgs, lib, ... }: {
pkgs,
lib,
config,
...
}:
let
runnerCfg = config.services.gitea-actions-runner;
instanceCfg = runnerCfg.instances.${instance};
# The same file upstream generates and hands the daemon.
runnerConfigFile = (pkgs.formats.yaml { }).generate "config.yaml" instanceCfg.settings;
# In the form the precond reads back from `.runner`.
registeredLabels = lib.concatStringsSep "," (lib.sort lib.lessThan instanceCfg.labels);
in
{ {
imports = [ imports = [
../container-modules/swarm-container.nix ../container-modules/swarm-container.nix
@ -321,8 +336,9 @@ in
systemd.tmpfiles.rules = [ systemd.tmpfiles.rules = [
"d ${stateDir} 0700 root root -" "d ${stateDir} 0700 root root -"
# `EnvironmentFile=` fails the unit when the file is missing; an # `LoadCredential=` fails the unit when the file is missing; an
# empty one leaves `TOKEN` unset for the precond to report. # empty one lets a registered runner start before any token has
# arrived, and the register step report the missing token.
"f ${tokenFile} 0600 root root -" "f ${tokenFile} 0600 root root -"
]; ];
@ -332,6 +348,8 @@ in
enable = true; enable = true;
inherit (cfg) name labels; inherit (cfg) name labels;
url = forgeUrl; url = forgeUrl;
# Upstream refuses an instance without a token source, and turns
# this one into the `EnvironmentFile=` the unit below drops.
inherit tokenFile; inherit tokenFile;
settings = { settings = {
runner.capacity = cfg.concurrency; runner.capacity = cfg.concurrency;
@ -360,54 +378,80 @@ in
environment = lib.mkIf caTrust.useSelfSigned { environment = lib.mkIf caTrust.useSelfSigned {
NODE_EXTRA_CA_CERTS = caTrust.caContainerPath; NODE_EXTRA_CA_CERTS = caTrust.caContainerPath;
}; };
# Runs before upstream's pre-start step, which registers whenever # The registration token reaches the unit only as a credential,
# `.runner` is absent or the token or labels changed. This one # which the register step reads. A host job inherits the daemon's
# removes `.runner` in the two cases upstream cannot see, and holds # whole environment, so neither carries it.
# the unit down until there is a token. serviceConfig.EnvironmentFile = lib.mkForce [ ];
# serviceConfig.LoadCredential = [ "${tokenCredential}:${tokenFile}" ];
# `+`: root, because `.runner`, the token and the system journal # One registration path: upstream's pre-start step registers from
# are all out of the dynamic user's reach. # `$TOKEN`, so this list replaces it.
serviceConfig.ExecStartPre = lib.mkMerge [ serviceConfig.ExecStartPre = lib.mkForce [
(lib.mkBefore [ # Removes `.runner` in the cases where reusing it would fail, so
"+${pkgs.writeShellScript "swarm-ci-runner-precond" '' # the register step after it registers again.
set -eu #
runner=${runnerFile} # `+`: root, because the system journal is out of the dynamic
started=${daemonStartedFile} # user's reach.
"+${pkgs.writeShellScript "swarm-ci-runner-precond" ''
set -eu
runner=${runnerFile}
started=${daemonStartedFile}
# The forge refused the registration the daemon last started # The forge refused the registration the daemon last started
# with. Read on the next start rather than when the daemon # with. Read on the next start rather than when the daemon
# exits, so journald has its last lines by then. # exits, so journald has its last lines by then.
if [ -f "$started" ]; then if [ -f "$started" ]; then
since="$(< "$started")" since="$(< "$started")"
${pkgs.coreutils}/bin/rm -f "$started" ${pkgs.coreutils}/bin/rm -f "$started"
if ${pkgs.systemd}/bin/journalctl -u ${runnerUnit}.service --since "@$since" -o cat --no-pager \ if ${pkgs.systemd}/bin/journalctl -u ${runnerUnit}.service --since "@$since" -o cat --no-pager \
| ${pkgs.gnugrep}/bin/grep -qF ${lib.escapeShellArg rejectedMessage}; then | ${pkgs.gnugrep}/bin/grep -qF ${lib.escapeShellArg rejectedMessage}; then
echo "the forge has no record of this runner's registration; registering again" >&2 echo "the forge has no record of this runner's registration; registering again" >&2
${pkgs.coreutils}/bin/rm -f "$runner"
fi
fi
# `.runner` records the address it registered at, and the
# daemon dials that one, not the configured one.
if [ -f "$runner" ] \
&& [ "$(${pkgs.jq}/bin/jq -r .address "$runner")" != ${lib.escapeShellArg forgeUrl} ]; then
echo "$runner names another forge address; registering again at ${forgeUrl}" >&2
${pkgs.coreutils}/bin/rm -f "$runner" ${pkgs.coreutils}/bin/rm -f "$runner"
fi fi
fi
# Upstream registers on a changed token hash, an empty token # `.runner` records the address it registered at, and the
# included, and a registration with no token fails. # daemon dials that one, not the configured one.
if [ -z "''${TOKEN:-}" ]; then if [ -f "$runner" ] \
echo "no registration token yet; swarm-bao-ci-runner on the host delivers it" >&2 && [ "$(${pkgs.jq}/bin/jq -r .address "$runner")" != ${lib.escapeShellArg forgeUrl} ]; then
exit 1 echo "$runner names another forge address; registering again at ${forgeUrl}" >&2
fi ${pkgs.coreutils}/bin/rm -f "$runner"
''}" fi
])
(lib.mkAfter [ # The daemon declares the labels `.runner` records, not the
"+${pkgs.writeShellScript "swarm-ci-runner-mark-start" '' # configured ones.
${pkgs.coreutils}/bin/date +%s > ${daemonStartedFile} if [ -f "$runner" ] \
''}" && [ "$(${pkgs.jq}/bin/jq -r '.labels | sort | join(",")' "$runner")" != ${lib.escapeShellArg registeredLabels} ]; then
]) echo "$runner names other labels; registering again with ${registeredLabels}" >&2
${pkgs.coreutils}/bin/rm -f "$runner"
fi
''}"
# Registers only when `.runner` is absent. As the dynamic user,
# so `.runner` is the daemon's own. The derivation itself rather
# than its path, so ../module-eval reads its text without building.
(pkgs.writeShellScript "swarm-ci-runner-register" ''
set -eu
dir="$STATE_DIRECTORY/${instance}"
${pkgs.coreutils}/bin/mkdir -p "$dir"
cd "$dir"
if [ -e .runner ]; then
exit 0
fi
token="$(< "$CREDENTIALS_DIRECTORY/${tokenCredential}")"
if [ -z "$token" ]; then
echo "no registration token yet; swarm-bao-ci-runner on the host delivers it" >&2
exit 1
fi
exec ${lib.getExe runnerCfg.package} register --no-interactive \
--instance ${lib.escapeShellArg instanceCfg.url} \
--token "$token" \
--name ${lib.escapeShellArg instanceCfg.name} \
--labels ${lib.escapeShellArg (lib.concatStringsSep "," instanceCfg.labels)} \
--config ${runnerConfigFile}
'')
"+${pkgs.writeShellScript "swarm-ci-runner-mark-start" ''
${pkgs.coreutils}/bin/date +%s > ${daemonStartedFile}
''}"
]; ];
# Restarted until the token arrives, however long the store and # Restarted until the token arrives, however long the store and
# the controller take. `mkForce`: upstream sets both. # the controller take. `mkForce`: upstream sets both.

34
nix/module-eval/forge-placement.nix
View file

@ -81,6 +81,9 @@ let
forgeSettings = forgeHere.containers.hive-forge.config.services.forgejo.settings; forgeSettings = forgeHere.containers.hive-forge.config.services.forgejo.settings;
runnerService = ciAlone.containers.swarm-ci-runner.config.systemd.services.gitea-runner-swarm;
registerStep = lib.findFirst lib.isDerivation null runnerService.serviceConfig.ExecStartPre;
cases = [ cases = [
{ {
# The absence the whole option exists for: a second forge in a swarm # The absence the whole option exists for: a second forge in a swarm
@ -150,6 +153,37 @@ let
name = "the CI runner is refused without its store identity"; name = "the CI runner is refused without its store identity";
ok = refusedOver ciNoIdentity "deploy.bao.ciRunnerClientCertFile"; ok = refusedOver ciNoIdentity "deploy.bao.ciRunnerClientCertFile";
} }
{
# A host job inherits the daemon's whole environment, and a token there
# lets any workflow register runners against the forge.
name = "the CI runner daemon carries no registration token in its environment";
ok =
let
sc = runnerService.serviceConfig;
in
!(runnerService.environment ? TOKEN)
&& lib.toList (sc.EnvironmentFile or [ ]) == [ ]
&& !(lib.any (lib.hasPrefix "TOKEN=") (lib.toList (sc.Environment or [ ])))
&& sc.LoadCredential == [ "registration-token:/var/lib/swarm-ci-runner/registration-token" ];
}
{
# The only reader of the credential, and the only registration path.
name = "the CI runner registers from its credential, and only without a .runner";
ok =
let
text = registerStep.text;
beforeRegister = lib.head (lib.splitString "register --no-interactive" text);
in
registerStep != null
&& !(lib.any (
s: lib.hasInfix "gitea-register-runner" (toString s)
) runnerService.serviceConfig.ExecStartPre)
&& lib.hasInfix "if [ -e .runner ]; then\n exit 0\nfi" beforeRegister
&& lib.hasInfix ''token="$(< "$CREDENTIALS_DIRECTORY/registration-token")"'' beforeRegister
&& lib.hasInfix ''--token "$token"'' text
&& !(lib.hasInfix "export" text)
&& !(lib.hasInfix "TOKEN" text);
}
{ {
# A removed option evaluates to an assertion naming the replacement. # A removed option evaluates to an assertion naming the replacement.
name = "a CI option under deploy.forgejo names its replacement under deploy.ci-runner"; name = "a CI option under deploy.forgejo names its replacement under deploy.ci-runner";