vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-ci-runner: run the forge CI runner as a swarm service

Browse source 
The CI runner was a hive deploy module: hive-ci.nix ran it beside the
forge, and hive-c0re registered it through the forge's admin API with
the `core` token, writing the registration token through hive-priv. It
could only run on the forge's host, and it kept an admin-scoped forge
token on a hive.

Now the runner is a swarm service:

- swarm-ci-runner.nix runs it in a `swarm-ci-runner` container on the
  host that sets deploy.ci-runner.enable, reaching the forge by name.
  It needs neither the forge nor a hive there.
- swarm-controller reads the forge's runner registration token with its
  own admin token and writes it to swarm/services/ci-runner/
  registration-token whenever it differs from what it last wrote. Its
  grant on that path is create/update only.
- The runner's own leaf (CN swarm-ci-runner, role and policy of the same
  name) reads that one path. swarm-bao-ci-runner.service fetches it into
  the runner's EnvironmentFile and restarts the runner on a change.
- A pre-start step removes .runner when the daemon's journal since its
  last start holds Forgejo's "unregistered runner" refusal (the daemon
  exits 1 when its startup Declare gets it), or when .runner names
  another forge address; upstream's pre-start step then registers again.

Removed from the hive side: hive-ci.nix and deploy.forgejo.ci.*,
forge/ci_runner.rs, PrivRequest::RegisterCiRunner and its handler and
ReadWritePaths entry, InfraContainer::Ci with hivectl's --ci and the
dashboard's hive-ci journal entry, HYPERHIVE_FORGE_CI_ENABLED, and
read:admin,write:admin from CORE_TOKEN_SCOPES: nothing else in hive-c0re
calls /api/v1/admin/*. The old ci options, under both swarm.forge.ci and
deploy.forgejo.ci, now fail evaluation naming deploy.ci-runner.

The forge's half of the old ci.enable, the actions/checkout mirror and
DEFAULT_ACTIONS_URL = self, is deploy.forgejo.actionsFromSelf.

Closes #4954
This commit is contained in:
atlas 2026-10-05 02:25:05 +02:00 • committed by mara
commit e2da5509ae
60 changed files with 1088 additions and 1201 deletions
Download patch file Download diff file Expand all files Collapse all files

10
hive-c0re/src/dashboard/journal.rs
View file

@ -1,8 +1,8 @@
//! Journal-read endpoints for the dashboard.
//!
//! `GET /api/journal/{name}` reads a managed agent container's journal, OR
//! one of the four hive infra services (`hive-ci`, `hive-forge`,
//! `hive-gateway`, `hive-matrix` — [`hive_priv_sock::InfraContainer`] is the
//! one of the three hive infra services (`hive-forge`, `hive-gateway`,
//! `hive-matrix` — [`hive_priv_sock::InfraContainer`] is the
//! allowlist). A container's journal is a `journalctl -M` read, delegated
//! to the root helper since entering a machine needs privileges hive-c0re
//! doesn't have; `hive-gateway` is nginx on the host, so it reads host
@ -42,8 +42,8 @@ pub(super) struct JournalQuery {
/// container namespace and needs root — is delegated to hive-priv.
///
/// `name` is either a managed agent name (`iris`, optionally already
/// carrying the `h-` prefix) or one of the four infra names (`hive-ci` /
/// `hive-forge` / `hive-gateway` / `hive-matrix` — see
/// carrying the `h-` prefix) or one of the three infra names (`hive-forge` /
/// `hive-gateway` / `hive-matrix` — see
/// [`hive_priv_sock::InfraContainer`]). Infra targets don't run the
/// per-agent hive daemons, so `unit` is ignored for them — the whole
/// machine journal, or for the gateway the host journal filtered to its
@ -52,7 +52,7 @@ pub(super) struct JournalQuery {
get,
path = "/api/journal/{name}",
params(
("name" = String, Path, description = "agent name, or one of the four infra container names"),
("name" = String, Path, description = "agent name, or one of the three infra container names"),
JournalQuery,
),
responses(

266
hive-c0re/src/forge/ci_runner.rs
View file

@ -1,266 +0,0 @@
//! hive-ci Forgejo Actions runner registration, driven from hive-c0re.
//!
//! Moves the runner-registration forge round-trip OFF the container's
//! boot-critical path (it was a host-side `hive-ci-prefetch` oneshot that
//! gated `container@hive-ci` start — see `nix/host-modules/hive-ci.nix`).
//! hive-c0re holds the forge admin token; here it validates the runner's
//! existing credentials and, when they're absent or stale, mints a fresh
//! registration token and hands it to hive-priv, which writes it to the host
//! env-file the container bind-mounts read-only and restarts the in-container
//! runner. The admin token never enters the container — only the registration
//! token does, exactly as the prefetch did.
use anyhow::{Context as _, Result};
use serde_json::Value;
use url::Url;
use super::forge_http_base;
/// Host path to the hive-ci runner's persisted credentials. The container is
/// non-ephemeral, so `.runner` survives restarts; a present file with a sane
/// `id` means the runner is already registered.
const RUNNER_FILE: &str = "/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner";
/// Whether the operator enabled the CI runner. The nix module sets
/// `HYPERHIVE_FORGE_CI_ENABLED=1` on `hive-c0re.service` when
/// `services.hyperhive.deploy.forgejo.ci.enable` is on; absent means CI is off and
/// there is no hive-ci container to register a runner for.
fn ci_enabled() -> bool {
std::env::var("HYPERHIVE_FORGE_CI_ENABLED")
.is_ok_and(|v| v == "1" || v.eq_ignore_ascii_case("true"))
}
/// Ensure the hive-ci runner is registered against the forge. Best-effort:
/// every failure is logged and swallowed so it never aborts the startup
/// sweep ([`super::ensure_all`]). No-op when CI is disabled or the runner
/// already holds valid credentials (so a healthy runner is never restarted).
pub(super) async fn ensure_ci_runner_registered(core_token: &str) {
if !ci_enabled() {
return;
}
// Existing creds still valid on the forge, AND pointed at the forge
// address we are configured for → nothing to do.
if let Some(id) = existing_runner_id()
&& runner_address_current()
&& runner_valid(core_token, id).await
{
return;
}
// Absent or stale → mint a fresh registration token and hand it to
// hive-priv (root) to write the env-file + restart the runner.
match fetch_registration_token(core_token).await {
Ok(token) => {
if let Err(e) = crate::priv_client::register_ci_runner(&token).await {
tracing::warn!(error = ?e, "ci runner: hive-priv register_ci_runner failed");
crate::warnings::set_boot_warning(
"ci_runner_register",
"crit",
format!("ci runner: hive-priv register_ci_runner failed: {e}"),
);
} else {
tracing::info!("ci runner: registered hive-ci runner with a fresh token");
}
}
Err(e) => {
tracing::warn!(error = ?e, "ci runner: fetch registration token failed");
crate::warnings::set_boot_warning(
"ci_runner_fetch_token",
"crit",
format!("ci runner: fetch registration token failed: {e}"),
);
}
}
}
/// Parse the runner id from the persisted `.runner` JSON, if present and sane
/// (a `0` id is the gitea-runner "unregistered" sentinel).
fn existing_runner_id() -> Option<u64> {
let raw = std::fs::read_to_string(RUNNER_FILE).ok()?;
let json: Value = serde_json::from_str(&raw).ok()?;
let id = json.get("id")?.as_u64()?;
(id != 0).then_some(id)
}
/// Whether the persisted `.runner` still names the forge host we are
/// configured for.
///
/// `act_runner` records the `--instance` URL it was registered with and the
/// daemon reads the address from there and nowhere else — the nix option only
/// reaches `register`, and upstream re-registers on a changed token or labels,
/// never on a changed URL. So a forge rename leaves a runner dialling a name
/// its container's `extraHosts` no longer resolves, with every existing check
/// still green: the runner id is untouched, so `runner_valid` says yes.
///
/// Fail-safe in the same direction as [`runner_valid`]: anything we cannot
/// read or parse returns `true` (keep the existing creds). Only a *positive*
/// disagreement between two parseable hosts is treated as stale, because the
/// cost of a false stale is re-registering on every boot.
fn runner_address_current() -> bool {
let Ok(raw) = std::fs::read_to_string(RUNNER_FILE) else {
return true;
};
runner_address_matches(&raw, forge_http_base())
}
/// Host-only comparison of `.runner`'s `address` against the configured base.
///
/// Split from [`runner_address_current`] so it is testable without the
/// filesystem. Compares **hosts**, not whole URLs: both strings come from the
/// same nix expression (`http://${swarm.forge.domain}`, reaching the runner as
/// `instances.hive.url` and this daemon as `HIVE_FORGE_URL`), but scheme,
/// port and trailing slash are exactly the kind of cosmetic difference that
/// would otherwise re-register the runner on every boot.
fn runner_address_matches(raw: &str, configured: &str) -> bool {
let Some(address) = serde_json::from_str::<Value>(raw)
.ok()
.and_then(|j| j.get("address")?.as_str().map(str::to_owned))
else {
return true;
};
match (Url::parse(&address), Url::parse(configured)) {
(Ok(a), Ok(b)) => match (a.host_str(), b.host_str()) {
(Some(a), Some(b)) => a.eq_ignore_ascii_case(b),
_ => true,
},
_ => true,
}
}
/// Bound on reaching the forge.
const HTTP_CONNECT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
/// Bound on one whole forge admin call, body included.
const HTTP_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(15);
/// A client with both forge bounds applied.
fn http_client() -> Result<reqwest::Client> {
reqwest::Client::builder()
.connect_timeout(HTTP_CONNECT_TIMEOUT)
.timeout(HTTP_TIMEOUT)
.build()
.context("build HTTP client")
}
/// Context for `what` failing with `e`; a timeout names the bound that fired.
fn http_context(what: &str, e: &reqwest::Error) -> String {
if e.is_connect() && e.is_timeout() {
format!("{what}: connect timed out after {HTTP_CONNECT_TIMEOUT:?}")
} else if e.is_timeout() {
format!("{what}: timed out after {HTTP_TIMEOUT:?}")
} else {
what.to_owned()
}
}
/// `GET /admin/runners/{id}` — `true` iff the runner still exists on the forge
/// (HTTP 200). A 404 (deleted from the admin panel) or any other status means
/// re-registration is needed. A transport error (forge unreachable) is treated
/// as "keep the existing creds" so a network blip never wipes a valid runner.
async fn runner_valid(core_token: &str, id: u64) -> bool {
let url = format!("{}/api/v1/admin/runners/{id}", forge_http_base());
let client = match http_client() {
Ok(c) => c,
Err(e) => {
tracing::warn!(error = ?e, "ci runner: build HTTP client failed; keeping existing creds");
return true;
}
};
match client
.get(&url)
.header("Authorization", format!("token {core_token}"))
.send()
.await
{
Ok(resp) => resp.status().is_success(),
Err(e) => {
let what = http_context("validation request failed", &e);
tracing::warn!(error = ?e, "ci runner: {what}; keeping existing creds");
true
}
}
}
/// `GET /admin/runners/registration-token` — mint a fresh registration token.
/// forgejo-api (0.11) doesn't wrap this endpoint, so this is a raw request
/// against the local HTTP forge ([`forge_http_base`]), mirroring the prefetch's
/// former curl; Forgejo accepts `Authorization: token <admin>`.
async fn fetch_registration_token(core_token: &str) -> Result<String> {
let url = format!(
"{}/api/v1/admin/runners/registration-token",
forge_http_base()
);
let resp = http_client()?
.get(&url)
.header("Authorization", format!("token {core_token}"))
.send()
.await
.map_err(|e| {
let what = http_context("GET admin/runners/registration-token", &e);
anyhow::Error::new(e).context(what)
})?;
let status = resp.status();
let json: Value = resp.json().await.map_err(|e| {
let what = http_context("parse registration-token response", &e);
anyhow::Error::new(e).context(what)
})?;
if !status.is_success() {
anyhow::bail!("registration-token HTTP {status}: {json}");
}
json.get("token")
.and_then(Value::as_str)
.map(str::to_owned)
.context("registration-token response missing 'token' field")
}
#[cfg(test)]
mod tests {
use super::runner_address_matches;
fn runner(address: &str) -> String {
format!(r#"{{"id":7,"uuid":"u","name":"hive","token":"t","address":"{address}"}}"#)
}
/// The case the bug is about: a forge rename leaves the old host behind.
#[test]
fn differing_host_is_stale() {
assert!(!runner_address_matches(
&runner("http://forge.pr1ma.darkest.space"),
"http://forge.constellation.darkest.space",
));
}
/// Control for the arm above — without it, a helper that always returned
/// `false` would pass `differing_host_is_stale` and re-register forever.
#[test]
fn same_host_is_current() {
assert!(runner_address_matches(
&runner("http://forge.example"),
"http://forge.example",
));
}
/// Cosmetic differences must NOT count: both sides are rendered from one
/// nix expression, but a port or trailing slash appearing on one side is
/// not a rename.
#[test]
fn port_scheme_and_trailing_slash_do_not_count() {
assert!(runner_address_matches(
&runner("http://forge.example:80/"),
"https://FORGE.example",
));
}
/// Fail-safe: anything unreadable keeps the existing credentials, so a
/// malformed or older `.runner` never costs a working runner.
#[test]
fn unparseable_input_keeps_creds() {
assert!(runner_address_matches("not json", "http://forge.example"));
assert!(runner_address_matches(
r#"{"id":7}"#,
"http://forge.example"
));
assert!(runner_address_matches(
&runner("://nonsense"),
"http://forge.example",
));
}
}

8
hive-c0re/src/forge/mod.rs
View file

@ -4,7 +4,6 @@
//! teams and repos).
//! No-op when `hive-forge` isn't running. Full design: `docs/integrations/forge.md`.
mod ci_runner;
mod pr_comment;
mod reconcile;
mod repos;
@ -233,8 +232,8 @@ pub async fn sync_agent(name: &str, core_token: Option<&str>) -> bool {
}
/// The `core_token.is_some()` half of [`ensure_all`]: the meta repo, the
/// local knowledge clone, the core avatar, CI runner registration, and
/// removing this hive's own config-PR hook —
/// local knowledge clone, the core avatar, and removing this hive's own
/// config-PR hook —
/// every step that needs an authenticated forge client. The swarm-wide
/// objects (orgs, the `operators` team, mirrors, `internal/docs`,
/// `internal/knowledge`, the `agent-configs` avatar) are the
@ -271,9 +270,6 @@ async fn ensure_all_orgs_and_repos(token: &str) {
format!("forge: core avatar upload failed: {e}"),
);
}
// Register the hive-ci Actions runner (off the container's boot path;
// no-op when CI is disabled or the runner already holds valid creds).
ci_runner::ensure_ci_runner_registered(token).await;
if let Err(e) = remove_config_pr_webhook(token).await {
tracing::warn!(error = ?e, "forge: remove_config_pr_webhook failed");
}

19
hive-c0re/src/forge/users.rs
View file

@ -38,13 +38,10 @@ fn core_avatar_png_path() -> std::path::PathBuf {
std::path::PathBuf::from(dir).join("branding/hyperhive.png")
}
/// Bootstrap `core` token scopes — adds `read:admin,write:admin` on
/// top of the agent scopes (swarm-controller's `AGENT_TOKEN_SCOPES`)
/// so the host daemon can drive `/api/v1/admin/*`. Site-admin
/// membership alone isn't enough: the token's own scope gate runs
/// before the user-permission check.
/// See `docs/integrations/forge.md::Token scopes`.
const CORE_TOKEN_SCOPES: &str = "read:admin,write:admin,read:user,write:user,read:notification,write:notification,write:repository,write:issue,write:organization,write:misc";
/// Bootstrap `core` token scopes — the same set as swarm-controller's
/// `AGENT_TOKEN_SCOPES`, with no admin scope: nothing in hive-c0re calls
/// `/api/v1/admin/*`. See `docs/integrations/forge.md::Token scopes`.
const CORE_TOKEN_SCOPES: &str = "read:user,write:user,read:notification,write:notification,write:repository,write:issue,write:organization,write:misc";
/// Pull the access token out of forgejo's success message. Format
/// has shifted across versions (table form vs. "Access token was
@ -197,9 +194,7 @@ pub(super) async fn ensure_core_avatar(token: &str) -> Result<()> {
/// Outcome of probing whether the persisted core token still works
/// against the *current* forge. Existence on disk is not validity: a
/// token minted before a forge rebuild / re-provision is unknown to the
/// new forge's DB and 401s on every call — which silently breaks the
/// hive-ci runner-registration prefetch (it reads this same token to
/// fetch a runner registration token).
/// new forge's DB and 401s on every call.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum CoreTokenCheck {
/// Token authenticated successfully — keep using it.
@ -260,8 +255,8 @@ async fn check_core_token(token: &str) -> CoreTokenCheck {
/// Ensure the bootstrap `core` admin user + a token at
/// `CORE_TOKEN_PATH`. The token is what hive-c0re uses for forgejo
/// API calls (org creation, meta-repo push, and the hive-ci
/// runner-registration prefetch). Returns the token.
/// API calls (meta-repo push, collaborator grants, PR comments).
/// Returns the token.
///
/// Idempotent, but validity-aware: when a token file is already present
/// it is **probed against the current forge** before being trusted. A

2
hive-c0re/src/paths.rs
View file

@ -242,7 +242,7 @@ pub fn gateway_agents_conf() -> PathBuf {
/// (used in `Path::new` + user-facing `format!` messages), so it stays a
/// const rather than a `PathBuf` fn.
// nix: read at provisioning; the same literal is in
// `nix/host-modules/hive-c0re/default.nix` and `nix/host-modules/hive-ci.nix` — must match.
// `nix/host-modules/hive-c0re/default.nix` — must match.
pub const FORGE_CORE_TOKEN: &str = "/var/lib/hyperhive/forge-core-token";
/// `/run/hyperhive` — the runtime root (host admin socket + per-agent dirs).

21
hive-c0re/src/priv_client.rs
View file

@ -369,25 +369,8 @@ pub async fn set_agent_paused(agent_name: &str, paused: bool) -> Result<()> {
.await?)
}
/// Register the hive-ci Forgejo Actions runner: hand the freshly-minted
/// registration token to hive-priv, which writes it to the host-side
/// `/run/hive-ci/runner-token` env-file and restarts the in-container runner.
/// The forge admin token stays in hive-c0re; only the registration token
/// crosses to the (host-path) env-file the container bind-mounts read-only.
///
/// # Errors
/// Propagates the hive-priv call failure: the socket / IPC error, or the
/// root-side error when the token is rejected (empty or control characters),
/// the env-file write fails, or the runner restart exits non-zero.
pub async fn register_ci_runner(token: &str) -> Result<()> {
ok(call(&PrivRequest::RegisterCiRunner {
token: token.to_owned(),
})
.await?)
}
/// Start / stop a hive infrastructure service (`hive-ci`, `hive-gateway`,
/// `hive-forge`, `hive-matrix`) on the host via `systemctl <action> <unit>`,
/// Start / stop a hive infrastructure service (`hive-gateway`, `hive-forge`,
/// `hive-matrix`) on the host via `systemctl <action> <unit>`,
/// where the unit is derived root-side from the variant
/// (`container@<name>.service`, or `nginx.service` for the gateway). The
/// [`InfraContainer`] enum is the allowlist — hive-priv needs no name

3
hive-c0re/src/server.rs
View file

@ -883,9 +883,6 @@ async fn scoped_agents(scope: &LifecycleScope) -> Result<Vec<String>> {
fn scoped_infra(scope: &LifecycleScope) -> Vec<InfraContainer> {
let everything = scope.is_everything();
let mut out = Vec::new();
if everything || scope.ci {
out.push(InfraContainer::Ci);
}
if everything || scope.forge {
out.push(InfraContainer::Forge);
}