vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-ci-runner: run the forge CI runner as a swarm service

Browse source 
The CI runner was a hive deploy module: hive-ci.nix ran it beside the
forge, and hive-c0re registered it through the forge's admin API with
the `core` token, writing the registration token through hive-priv. It
could only run on the forge's host, and it kept an admin-scoped forge
token on a hive.

Now the runner is a swarm service:

- swarm-ci-runner.nix runs it in a `swarm-ci-runner` container on the
  host that sets deploy.ci-runner.enable, reaching the forge by name.
  It needs neither the forge nor a hive there.
- swarm-controller reads the forge's runner registration token with its
  own admin token and writes it to swarm/services/ci-runner/
  registration-token whenever it differs from what it last wrote. Its
  grant on that path is create/update only.
- The runner's own leaf (CN swarm-ci-runner, role and policy of the same
  name) reads that one path. swarm-bao-ci-runner.service fetches it into
  the runner's EnvironmentFile and restarts the runner on a change.
- A pre-start step removes .runner when the daemon's journal since its
  last start holds Forgejo's "unregistered runner" refusal (the daemon
  exits 1 when its startup Declare gets it), or when .runner names
  another forge address; upstream's pre-start step then registers again.

Removed from the hive side: hive-ci.nix and deploy.forgejo.ci.*,
forge/ci_runner.rs, PrivRequest::RegisterCiRunner and its handler and
ReadWritePaths entry, InfraContainer::Ci with hivectl's --ci and the
dashboard's hive-ci journal entry, HYPERHIVE_FORGE_CI_ENABLED, and
read:admin,write:admin from CORE_TOKEN_SCOPES: nothing else in hive-c0re
calls /api/v1/admin/*. The old ci options, under both swarm.forge.ci and
deploy.forgejo.ci, now fail evaluation naming deploy.ci-runner.

The forge's half of the old ci.enable, the actions/checkout mirror and
DEFAULT_ACTIONS_URL = self, is deploy.forgejo.actionsFromSelf.

Closes #4954
This commit is contained in:
atlas 2026-10-05 02:25:05 +02:00 • committed by mara
commit e2da5509ae
60 changed files with 1088 additions and 1201 deletions
Download patch file Download diff file Expand all files Collapse all files

168
hive-priv/src/main.rs
View file

@ -407,8 +407,6 @@ async fn exec(
paused,
} => exec_set_agent_paused(agent_name, paused),
PrivRequest::RegisterCiRunner { ref token } => register_ci_runner(token).await,
PrivRequest::ControlInfraContainer { container, action } => {
control_infra_container(container, action).await
}
@ -1167,107 +1165,6 @@ async fn daemon_reload() -> Result<(String, String)> {
Ok((String::new(), String::new()))
}
/// Host path to the hive-ci runner's persisted registration credentials.
///
/// Paired with `hive-c0re`'s `forge::ci_runner::RUNNER_FILE`, which reads the
/// same file to decide whether a runner is registered and whether it still
/// names the configured forge host. Deliberately duplicated rather than shared:
/// `hive-priv` is the minimal root helper and does not depend on `hive-c0re`.
const RUNNER_CREDENTIALS: &str =
"/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner";
/// Delete the runner's persisted credentials so upstream's `ExecStartPre` takes
/// its **absence** branch on the next start.
///
/// Absence is the state we want, so `NotFound` is success. Anything else — a
/// permission error above all — is NOT swallowed: it means the file is still
/// there, the restart will take upstream's already-registered branch, and the
/// caller would return `Ok` for a registration that never happened. That is the
/// same shape as a precondition that "passes" because it could not read the file
/// it was checking, and it is worth failing loudly to avoid.
///
/// Split from [`register_ci_runner`] purely so this rule is testable without a
/// container or a `systemctl`.
fn clear_runner_credentials(path: &str) -> Result<()> {
match std::fs::remove_file(path) {
Ok(()) => Ok(()),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(e) => {
Err(anyhow::Error::new(e).context(format!("remove stale runner credentials {path}")))
}
}
}
/// `RegisterCiRunner` — write the runner registration token to the host-side
/// `/run/hive-ci/runner-token` env-file, then restart the in-container runner
/// so it re-registers. The forge admin token never enters the container; only
/// the registration token c0re passes here is written, and it lands on a host
/// path bind-mounted read-only into hive-ci.
async fn register_ci_runner(token: &str) -> Result<(String, String)> {
use std::io::Write as _;
use std::os::unix::fs::{OpenOptionsExt as _, PermissionsExt as _};
// Reject anything that could corrupt the `KEY=VALUE` env-file or smuggle a
// second line — a forge registration token is an opaque single-line string.
if token.is_empty() || token.contains(['\n', '\r', '\0']) {
bail!("ci runner registration token empty or contains control characters");
}
let token_path = "/run/hive-ci/runner-token";
// In-place truncate+write of the existing inode (mirrors the prefetch's
// `echo > $FILE`), NOT a temp+rename: nspawn pins this file's inode into
// hive-ci at container start, so a rename would leave the running runner
// reading the old content. Format + perms match the tmpfiles seed and the
// prefetch: `TOKEN=<tok>`, mode 0600, root-owned. Created 0600 when the
// seed is missing, so the token is never on disk at a wider mode.
std::fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.mode(0o600)
.open(token_path)
.and_then(|mut f| f.write_all(format!("TOKEN={token}\n").as_bytes()))
.with_context(|| format!("write {token_path}"))?;
std::fs::set_permissions(token_path, std::fs::Permissions::from_mode(0o600))
.with_context(|| format!("chmod {token_path}"))?;
// Remove the persisted credentials BEFORE restarting, or the restart is a
// no-op as far as registration goes.
//
// Upstream's `ExecStartPre` only re-registers when `.runner` is absent, the
// labels changed, or the *registration token hash* changed — never when the
// instance URL changed. c0re only calls this helper once it has already
// decided the existing credentials are absent or stale
// (`forge::ci_runner::ensure_ci_runner_registered` returns early otherwise),
// so by the time we are here a re-registration is exactly what is wanted and
// deleting the file is the narrow way to guarantee it happens.
//
// Writing a fresh token is NOT sufficient on its own: whether the hash
// changes depends on whether the forge mints a new registration token per
// request or hands back a stable one, which is Forgejo's behaviour to
// choose and change. Gating our remediation on the absence branch — the one
// upstream evaluates unconditionally — makes that question moot instead of
// load-bearing.
//
// See [`clear_runner_credentials`] for why absence is the branch we aim at
// and why only `NotFound` counts as success.
clear_runner_credentials(RUNNER_CREDENTIALS)?;
// Restart the in-container runner so it reads the new token and registers.
let out = Command::new("systemctl")
.args(["--machine=hive-ci", "restart", "gitea-runner-hive.service"])
.output()
.await
.context("systemctl restart gitea-runner-hive.service in hive-ci")?;
if !out.status.success() {
bail!(
"systemctl restart gitea-runner-hive.service in hive-ci exited {}: {}",
out.status,
String::from_utf8_lossy(&out.stderr).trim()
);
}
Ok((
String::from_utf8_lossy(&out.stdout).into_owned(),
String::from_utf8_lossy(&out.stderr).into_owned(),
))
}
/// `ControlInfraContainer` — start/stop a hive infrastructure service via
/// `systemctl <verb> <unit>`. The [`InfraContainer`] enum is the allowlist:
/// serde already rejected any unknown / unsafe name (hive-c0re has no
@ -3196,12 +3093,11 @@ fn ensure_socket_dir_in(root: &Path, name: &str) -> Result<()> {
mod tests {
use super::{
BindMount, BoundedRun, OwnedFd, PAUSED_MARKER_FILE, PrivRequest, StagedFile,
check_fd_agreement, clear_runner_credentials, contains_secret_shaped_run,
describe_forge_admin, ensure_plain_filename, ensure_socket_dir_in, git_overlay_flags,
limits_dropin_body, open_dir, open_export_dest, partial_name, publish_file,
redact_secret_line, remove_marker_in, run_bounded, single_output_path, toplevel_attr,
validate_credential_name, validate_snapshot_name, write_agent_dir_file,
write_bridge_dns_marker_in,
check_fd_agreement, contains_secret_shaped_run, describe_forge_admin,
ensure_plain_filename, ensure_socket_dir_in, git_overlay_flags, limits_dropin_body,
open_dir, open_export_dest, partial_name, publish_file, redact_secret_line,
remove_marker_in, run_bounded, single_output_path, toplevel_attr, validate_credential_name,
validate_snapshot_name, write_agent_dir_file, write_bridge_dns_marker_in,
};
use std::path::PathBuf;
use std::sync::atomic::{AtomicU32, Ordering};
@ -4035,60 +3931,6 @@ mod tests {
assert!(serde_json::from_str::<PrivRequest>(r#"{"op":"sync_agent_tmpfilez"}"#).is_err());
}
/// The runner-credential clear, in all three states that matter. The
/// PRESENCE arm is the load-bearing one: an implementation that did nothing
/// at all would pass the "absent is fine" arm perfectly, and the whole point
/// of the call is that the file is *gone* afterwards — upstream re-registers
/// on absence and on nothing else we control.
#[test]
fn clearing_runner_credentials_removes_it_and_tolerates_absence() {
let dir = scratch();
let path = dir.join(".runner");
let path_str = path.to_str().unwrap();
// Absent → success (this is the state we are aiming for).
assert!(!path.exists());
clear_runner_credentials(path_str).unwrap();
// Present → success AND actually gone. Without this arm a no-op passes.
std::fs::write(&path, r#"{"id":7,"address":"http://old.invalid"}"#).unwrap();
assert!(
path.exists(),
"control: the file must exist before the clear"
);
clear_runner_credentials(path_str).unwrap();
assert!(
!path.exists(),
"stale credentials must be GONE, or the restart takes upstream's \
already-registered branch and registration silently never happens"
);
std::fs::remove_dir_all(&dir).ok();
}
/// A failure that is not `NotFound` must propagate, never read as success.
/// A directory in the file's place makes `remove_file` fail with a non-
/// `NotFound` error without needing to drop privileges in a test.
#[test]
fn clearing_runner_credentials_propagates_a_real_failure() {
let dir = scratch();
let path = dir.join(".runner");
std::fs::create_dir(&path).unwrap();
let err = clear_runner_credentials(path.to_str().unwrap())
.expect_err("a non-NotFound failure must NOT be reported as success");
assert!(
format!("{err:#}").contains("remove stale runner credentials"),
"error must name what it failed to do, got: {err:#}"
);
assert!(
path.exists(),
"nothing was removed, and the caller must know"
);
std::fs::remove_dir_all(&dir).ok();
}
/// The pause marker round-trips through the same root-only path the
/// credential writes use, and BOTH directions are idempotent — the
/// dashboard toggle and `hivectl pause|resume` fire blind, without