swarm-ci-runner: run the forge CI runner as a swarm service
The CI runner was a hive deploy module: hive-ci.nix ran it beside the forge, and hive-c0re registered it through the forge's admin API with the `core` token, writing the registration token through hive-priv. It could only run on the forge's host, and it kept an admin-scoped forge token on a hive. Now the runner is a swarm service: - swarm-ci-runner.nix runs it in a `swarm-ci-runner` container on the host that sets deploy.ci-runner.enable, reaching the forge by name. It needs neither the forge nor a hive there. - swarm-controller reads the forge's runner registration token with its own admin token and writes it to swarm/services/ci-runner/ registration-token whenever it differs from what it last wrote. Its grant on that path is create/update only. - The runner's own leaf (CN swarm-ci-runner, role and policy of the same name) reads that one path. swarm-bao-ci-runner.service fetches it into the runner's EnvironmentFile and restarts the runner on a change. - A pre-start step removes .runner when the daemon's journal since its last start holds Forgejo's "unregistered runner" refusal (the daemon exits 1 when its startup Declare gets it), or when .runner names another forge address; upstream's pre-start step then registers again. Removed from the hive side: hive-ci.nix and deploy.forgejo.ci.*, forge/ci_runner.rs, PrivRequest::RegisterCiRunner and its handler and ReadWritePaths entry, InfraContainer::Ci with hivectl's --ci and the dashboard's hive-ci journal entry, HYPERHIVE_FORGE_CI_ENABLED, and read:admin,write:admin from CORE_TOKEN_SCOPES: nothing else in hive-c0re calls /api/v1/admin/*. The old ci options, under both swarm.forge.ci and deploy.forgejo.ci, now fail evaluation naming deploy.ci-runner. The forge's half of the old ci.enable, the actions/checkout mirror and DEFAULT_ACTIONS_URL = self, is deploy.forgejo.actionsFromSelf. Closes #4954
This commit is contained in:
atlas
mara
parent
3d43e9588c
commit
e2da5509ae
60 changed files with 1088 additions and 1201 deletions
|
|
@ -1,266 +0,0 @@
|
|||
//! hive-ci Forgejo Actions runner registration, driven from hive-c0re.
|
||||
//!
|
||||
//! Moves the runner-registration forge round-trip OFF the container's
|
||||
//! boot-critical path (it was a host-side `hive-ci-prefetch` oneshot that
|
||||
//! gated `container@hive-ci` start — see `nix/host-modules/hive-ci.nix`).
|
||||
//! hive-c0re holds the forge admin token; here it validates the runner's
|
||||
//! existing credentials and, when they're absent or stale, mints a fresh
|
||||
//! registration token and hands it to hive-priv, which writes it to the host
|
||||
//! env-file the container bind-mounts read-only and restarts the in-container
|
||||
//! runner. The admin token never enters the container — only the registration
|
||||
//! token does, exactly as the prefetch did.
|
||||
|
||||
use anyhow::{Context as _, Result};
|
||||
use serde_json::Value;
|
||||
use url::Url;
|
||||
|
||||
use super::forge_http_base;
|
||||
|
||||
/// Host path to the hive-ci runner's persisted credentials. The container is
|
||||
/// non-ephemeral, so `.runner` survives restarts; a present file with a sane
|
||||
/// `id` means the runner is already registered.
|
||||
const RUNNER_FILE: &str = "/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner";
|
||||
|
||||
/// Whether the operator enabled the CI runner. The nix module sets
|
||||
/// `HYPERHIVE_FORGE_CI_ENABLED=1` on `hive-c0re.service` when
|
||||
/// `services.hyperhive.deploy.forgejo.ci.enable` is on; absent means CI is off and
|
||||
/// there is no hive-ci container to register a runner for.
|
||||
fn ci_enabled() -> bool {
|
||||
std::env::var("HYPERHIVE_FORGE_CI_ENABLED")
|
||||
.is_ok_and(|v| v == "1" || v.eq_ignore_ascii_case("true"))
|
||||
}
|
||||
|
||||
/// Ensure the hive-ci runner is registered against the forge. Best-effort:
|
||||
/// every failure is logged and swallowed so it never aborts the startup
|
||||
/// sweep ([`super::ensure_all`]). No-op when CI is disabled or the runner
|
||||
/// already holds valid credentials (so a healthy runner is never restarted).
|
||||
pub(super) async fn ensure_ci_runner_registered(core_token: &str) {
|
||||
if !ci_enabled() {
|
||||
return;
|
||||
}
|
||||
// Existing creds still valid on the forge, AND pointed at the forge
|
||||
// address we are configured for → nothing to do.
|
||||
if let Some(id) = existing_runner_id()
|
||||
&& runner_address_current()
|
||||
&& runner_valid(core_token, id).await
|
||||
{
|
||||
return;
|
||||
}
|
||||
// Absent or stale → mint a fresh registration token and hand it to
|
||||
// hive-priv (root) to write the env-file + restart the runner.
|
||||
match fetch_registration_token(core_token).await {
|
||||
Ok(token) => {
|
||||
if let Err(e) = crate::priv_client::register_ci_runner(&token).await {
|
||||
tracing::warn!(error = ?e, "ci runner: hive-priv register_ci_runner failed");
|
||||
crate::warnings::set_boot_warning(
|
||||
"ci_runner_register",
|
||||
"crit",
|
||||
format!("ci runner: hive-priv register_ci_runner failed: {e}"),
|
||||
);
|
||||
} else {
|
||||
tracing::info!("ci runner: registered hive-ci runner with a fresh token");
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(error = ?e, "ci runner: fetch registration token failed");
|
||||
crate::warnings::set_boot_warning(
|
||||
"ci_runner_fetch_token",
|
||||
"crit",
|
||||
format!("ci runner: fetch registration token failed: {e}"),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse the runner id from the persisted `.runner` JSON, if present and sane
|
||||
/// (a `0` id is the gitea-runner "unregistered" sentinel).
|
||||
fn existing_runner_id() -> Option<u64> {
|
||||
let raw = std::fs::read_to_string(RUNNER_FILE).ok()?;
|
||||
let json: Value = serde_json::from_str(&raw).ok()?;
|
||||
let id = json.get("id")?.as_u64()?;
|
||||
(id != 0).then_some(id)
|
||||
}
|
||||
|
||||
/// Whether the persisted `.runner` still names the forge host we are
|
||||
/// configured for.
|
||||
///
|
||||
/// `act_runner` records the `--instance` URL it was registered with and the
|
||||
/// daemon reads the address from there and nowhere else — the nix option only
|
||||
/// reaches `register`, and upstream re-registers on a changed token or labels,
|
||||
/// never on a changed URL. So a forge rename leaves a runner dialling a name
|
||||
/// its container's `extraHosts` no longer resolves, with every existing check
|
||||
/// still green: the runner id is untouched, so `runner_valid` says yes.
|
||||
///
|
||||
/// Fail-safe in the same direction as [`runner_valid`]: anything we cannot
|
||||
/// read or parse returns `true` (keep the existing creds). Only a *positive*
|
||||
/// disagreement between two parseable hosts is treated as stale, because the
|
||||
/// cost of a false stale is re-registering on every boot.
|
||||
fn runner_address_current() -> bool {
|
||||
let Ok(raw) = std::fs::read_to_string(RUNNER_FILE) else {
|
||||
return true;
|
||||
};
|
||||
runner_address_matches(&raw, forge_http_base())
|
||||
}
|
||||
|
||||
/// Host-only comparison of `.runner`'s `address` against the configured base.
|
||||
///
|
||||
/// Split from [`runner_address_current`] so it is testable without the
|
||||
/// filesystem. Compares **hosts**, not whole URLs: both strings come from the
|
||||
/// same nix expression (`http://${swarm.forge.domain}`, reaching the runner as
|
||||
/// `instances.hive.url` and this daemon as `HIVE_FORGE_URL`), but scheme,
|
||||
/// port and trailing slash are exactly the kind of cosmetic difference that
|
||||
/// would otherwise re-register the runner on every boot.
|
||||
fn runner_address_matches(raw: &str, configured: &str) -> bool {
|
||||
let Some(address) = serde_json::from_str::<Value>(raw)
|
||||
.ok()
|
||||
.and_then(|j| j.get("address")?.as_str().map(str::to_owned))
|
||||
else {
|
||||
return true;
|
||||
};
|
||||
match (Url::parse(&address), Url::parse(configured)) {
|
||||
(Ok(a), Ok(b)) => match (a.host_str(), b.host_str()) {
|
||||
(Some(a), Some(b)) => a.eq_ignore_ascii_case(b),
|
||||
_ => true,
|
||||
},
|
||||
_ => true,
|
||||
}
|
||||
}
|
||||
|
||||
/// Bound on reaching the forge.
|
||||
const HTTP_CONNECT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
|
||||
/// Bound on one whole forge admin call, body included.
|
||||
const HTTP_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(15);
|
||||
|
||||
/// A client with both forge bounds applied.
|
||||
fn http_client() -> Result<reqwest::Client> {
|
||||
reqwest::Client::builder()
|
||||
.connect_timeout(HTTP_CONNECT_TIMEOUT)
|
||||
.timeout(HTTP_TIMEOUT)
|
||||
.build()
|
||||
.context("build HTTP client")
|
||||
}
|
||||
|
||||
/// Context for `what` failing with `e`; a timeout names the bound that fired.
|
||||
fn http_context(what: &str, e: &reqwest::Error) -> String {
|
||||
if e.is_connect() && e.is_timeout() {
|
||||
format!("{what}: connect timed out after {HTTP_CONNECT_TIMEOUT:?}")
|
||||
} else if e.is_timeout() {
|
||||
format!("{what}: timed out after {HTTP_TIMEOUT:?}")
|
||||
} else {
|
||||
what.to_owned()
|
||||
}
|
||||
}
|
||||
|
||||
/// `GET /admin/runners/{id}` — `true` iff the runner still exists on the forge
|
||||
/// (HTTP 200). A 404 (deleted from the admin panel) or any other status means
|
||||
/// re-registration is needed. A transport error (forge unreachable) is treated
|
||||
/// as "keep the existing creds" so a network blip never wipes a valid runner.
|
||||
async fn runner_valid(core_token: &str, id: u64) -> bool {
|
||||
let url = format!("{}/api/v1/admin/runners/{id}", forge_http_base());
|
||||
let client = match http_client() {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = ?e, "ci runner: build HTTP client failed; keeping existing creds");
|
||||
return true;
|
||||
}
|
||||
};
|
||||
match client
|
||||
.get(&url)
|
||||
.header("Authorization", format!("token {core_token}"))
|
||||
.send()
|
||||
.await
|
||||
{
|
||||
Ok(resp) => resp.status().is_success(),
|
||||
Err(e) => {
|
||||
let what = http_context("validation request failed", &e);
|
||||
tracing::warn!(error = ?e, "ci runner: {what}; keeping existing creds");
|
||||
true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `GET /admin/runners/registration-token` — mint a fresh registration token.
|
||||
/// forgejo-api (0.11) doesn't wrap this endpoint, so this is a raw request
|
||||
/// against the local HTTP forge ([`forge_http_base`]), mirroring the prefetch's
|
||||
/// former curl; Forgejo accepts `Authorization: token <admin>`.
|
||||
async fn fetch_registration_token(core_token: &str) -> Result<String> {
|
||||
let url = format!(
|
||||
"{}/api/v1/admin/runners/registration-token",
|
||||
forge_http_base()
|
||||
);
|
||||
let resp = http_client()?
|
||||
.get(&url)
|
||||
.header("Authorization", format!("token {core_token}"))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| {
|
||||
let what = http_context("GET admin/runners/registration-token", &e);
|
||||
anyhow::Error::new(e).context(what)
|
||||
})?;
|
||||
let status = resp.status();
|
||||
let json: Value = resp.json().await.map_err(|e| {
|
||||
let what = http_context("parse registration-token response", &e);
|
||||
anyhow::Error::new(e).context(what)
|
||||
})?;
|
||||
if !status.is_success() {
|
||||
anyhow::bail!("registration-token HTTP {status}: {json}");
|
||||
}
|
||||
json.get("token")
|
||||
.and_then(Value::as_str)
|
||||
.map(str::to_owned)
|
||||
.context("registration-token response missing 'token' field")
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::runner_address_matches;
|
||||
|
||||
fn runner(address: &str) -> String {
|
||||
format!(r#"{{"id":7,"uuid":"u","name":"hive","token":"t","address":"{address}"}}"#)
|
||||
}
|
||||
|
||||
/// The case the bug is about: a forge rename leaves the old host behind.
|
||||
#[test]
|
||||
fn differing_host_is_stale() {
|
||||
assert!(!runner_address_matches(
|
||||
&runner("http://forge.pr1ma.darkest.space"),
|
||||
"http://forge.constellation.darkest.space",
|
||||
));
|
||||
}
|
||||
|
||||
/// Control for the arm above — without it, a helper that always returned
|
||||
/// `false` would pass `differing_host_is_stale` and re-register forever.
|
||||
#[test]
|
||||
fn same_host_is_current() {
|
||||
assert!(runner_address_matches(
|
||||
&runner("http://forge.example"),
|
||||
"http://forge.example",
|
||||
));
|
||||
}
|
||||
|
||||
/// Cosmetic differences must NOT count: both sides are rendered from one
|
||||
/// nix expression, but a port or trailing slash appearing on one side is
|
||||
/// not a rename.
|
||||
#[test]
|
||||
fn port_scheme_and_trailing_slash_do_not_count() {
|
||||
assert!(runner_address_matches(
|
||||
&runner("http://forge.example:80/"),
|
||||
"https://FORGE.example",
|
||||
));
|
||||
}
|
||||
|
||||
/// Fail-safe: anything unreadable keeps the existing credentials, so a
|
||||
/// malformed or older `.runner` never costs a working runner.
|
||||
#[test]
|
||||
fn unparseable_input_keeps_creds() {
|
||||
assert!(runner_address_matches("not json", "http://forge.example"));
|
||||
assert!(runner_address_matches(
|
||||
r#"{"id":7}"#,
|
||||
"http://forge.example"
|
||||
));
|
||||
assert!(runner_address_matches(
|
||||
&runner("://nonsense"),
|
||||
"http://forge.example",
|
||||
));
|
||||
}
|
||||
}
|
||||
|
|
@ -4,7 +4,6 @@
|
|||
//! teams and repos).
|
||||
//! No-op when `hive-forge` isn't running. Full design: `docs/integrations/forge.md`.
|
||||
|
||||
mod ci_runner;
|
||||
mod pr_comment;
|
||||
mod reconcile;
|
||||
mod repos;
|
||||
|
|
@ -233,8 +232,8 @@ pub async fn sync_agent(name: &str, core_token: Option<&str>) -> bool {
|
|||
}
|
||||
|
||||
/// The `core_token.is_some()` half of [`ensure_all`]: the meta repo, the
|
||||
/// local knowledge clone, the core avatar, CI runner registration, and
|
||||
/// removing this hive's own config-PR hook —
|
||||
/// local knowledge clone, the core avatar, and removing this hive's own
|
||||
/// config-PR hook —
|
||||
/// every step that needs an authenticated forge client. The swarm-wide
|
||||
/// objects (orgs, the `operators` team, mirrors, `internal/docs`,
|
||||
/// `internal/knowledge`, the `agent-configs` avatar) are the
|
||||
|
|
@ -271,9 +270,6 @@ async fn ensure_all_orgs_and_repos(token: &str) {
|
|||
format!("forge: core avatar upload failed: {e}"),
|
||||
);
|
||||
}
|
||||
// Register the hive-ci Actions runner (off the container's boot path;
|
||||
// no-op when CI is disabled or the runner already holds valid creds).
|
||||
ci_runner::ensure_ci_runner_registered(token).await;
|
||||
if let Err(e) = remove_config_pr_webhook(token).await {
|
||||
tracing::warn!(error = ?e, "forge: remove_config_pr_webhook failed");
|
||||
}
|
||||
|
|
|
|||
|
|
@ -38,13 +38,10 @@ fn core_avatar_png_path() -> std::path::PathBuf {
|
|||
std::path::PathBuf::from(dir).join("branding/hyperhive.png")
|
||||
}
|
||||
|
||||
/// Bootstrap `core` token scopes — adds `read:admin,write:admin` on
|
||||
/// top of the agent scopes (swarm-controller's `AGENT_TOKEN_SCOPES`)
|
||||
/// so the host daemon can drive `/api/v1/admin/*`. Site-admin
|
||||
/// membership alone isn't enough: the token's own scope gate runs
|
||||
/// before the user-permission check.
|
||||
/// See `docs/integrations/forge.md::Token scopes`.
|
||||
const CORE_TOKEN_SCOPES: &str = "read:admin,write:admin,read:user,write:user,read:notification,write:notification,write:repository,write:issue,write:organization,write:misc";
|
||||
/// Bootstrap `core` token scopes — the same set as swarm-controller's
|
||||
/// `AGENT_TOKEN_SCOPES`, with no admin scope: nothing in hive-c0re calls
|
||||
/// `/api/v1/admin/*`. See `docs/integrations/forge.md::Token scopes`.
|
||||
const CORE_TOKEN_SCOPES: &str = "read:user,write:user,read:notification,write:notification,write:repository,write:issue,write:organization,write:misc";
|
||||
|
||||
/// Pull the access token out of forgejo's success message. Format
|
||||
/// has shifted across versions (table form vs. "Access token was
|
||||
|
|
@ -197,9 +194,7 @@ pub(super) async fn ensure_core_avatar(token: &str) -> Result<()> {
|
|||
/// Outcome of probing whether the persisted core token still works
|
||||
/// against the *current* forge. Existence on disk is not validity: a
|
||||
/// token minted before a forge rebuild / re-provision is unknown to the
|
||||
/// new forge's DB and 401s on every call — which silently breaks the
|
||||
/// hive-ci runner-registration prefetch (it reads this same token to
|
||||
/// fetch a runner registration token).
|
||||
/// new forge's DB and 401s on every call.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
enum CoreTokenCheck {
|
||||
/// Token authenticated successfully — keep using it.
|
||||
|
|
@ -260,8 +255,8 @@ async fn check_core_token(token: &str) -> CoreTokenCheck {
|
|||
|
||||
/// Ensure the bootstrap `core` admin user + a token at
|
||||
/// `CORE_TOKEN_PATH`. The token is what hive-c0re uses for forgejo
|
||||
/// API calls (org creation, meta-repo push, and the hive-ci
|
||||
/// runner-registration prefetch). Returns the token.
|
||||
/// API calls (meta-repo push, collaborator grants, PR comments).
|
||||
/// Returns the token.
|
||||
///
|
||||
/// Idempotent, but validity-aware: when a token file is already present
|
||||
/// it is **probed against the current forge** before being trusted. A
|
||||
|
|
|
|||
Loading…
Reference in a new issue