Compare commits

...
Author SHA1 Message Date
atlas
af7ec98542 doc: ensure_mirror_repo docstring — 409 only, not 409/422 (match the fix) 2026-06-29 00:26:41 +02:00
atlas
64e51fe3bf address argus: 422 from migrate is a validation error, not 'exists'
ensure_mirror_repo treated 409|422 as success (copied from ensure_org, where
422 *does* mean 'org exists'). For the migrate endpoint 422 is a validation
error (bad clone_addr/service); the GET-first check is the real idempotency
guard, so 409 stays as a race guard but 422 now falls through to the bail arm
(→ caller warns) instead of silently dropping a misconfigured mirror.
2026-06-29 00:26:41 +02:00
atlas
53df2c9598 rework(#2072): seed mirrors in c0re startup, not a host-side oneshot
Per mara: the mirror seeding belongs in hive-c0re's forge provisioning
sweep, where the core admin token + org-ensure already live — not a parallel
host-side nix oneshot.

- forge.rs: ensure_mirrors() reads HYPERHIVE_FORGE_MIRRORS (JSON list of
  {upstream,dest}), ensures each dest org (reuse ensure_org) + creates the
  pull-mirror via the migrate API (reuse forge_http, serde_json::json! body,
  409/existing = success). Called in ensure_all() right after the SEEDED_ORGS
  loop (token in scope, warn-and-continue like the other ensure_* steps).
- hive-forge.nix: forward effectiveMirrors to c0re via
  systemd.services.hive-c0re.environment.HYPERHIVE_FORGE_MIRRORS; drop the
  forgejo-seed-mirrors.service + its script + the host-side core-token read.
  Keep the forge.mirrors option, DEFAULT_ACTIONS_URL=self (CI-gated), and the
  dest-shape / no-c0re-namespace-collision assertions.

Verified locally: nix parse + treefmt (incl rustfmt) clean; serde/serde_json
patterns mirror dashboard.rs. cargo build runs in CI (no cc in my container).
2026-06-29 00:26:41 +02:00
atlas
6f5dade9c9 treefmt: collapse the assertion-message interpolation to one line
nixfmt wanted the ${...} on a single line (verified locally:
nix build .#checks.x86_64-linux.formatting passes). The earlier CI 'docs'
failure was a transient muede-pc2 build hiccup — the docs check builds clean
locally on the same drv.
2026-06-29 00:26:41 +02:00
atlas
990868b2e7 address argus review on the mirror seed
- drop the three cross-ref tracker tags from comments/description (prose only)
- build the orgs + migrate JSON bodies with jq -n --arg (an upstream URL
  containing a quote no longer corrupts the request)
- don't auto-append the actions/checkout mirror when the operator already
  declared that dest (avoids a duplicate effectiveMirrors entry when CI is on)
2026-06-29 00:26:41 +02:00
atlas
4a3581a3d2 feat(#2072): auto-seed Forgejo pull-mirrors (DEFAULT_ACTIONS_URL=self for CI)
General-purpose mirror mechanism for the internal forge, per mara's call on
#2074 (real Forgejo pull-mirrors, nix-configured — not a pushed clone).

- services.hyperhive.forge.mirrors: list of { upstream, dest } pull-mirrors,
  any repo. Each is created as a real Forgejo pull-mirror (re-syncs from
  upstream), dest = <owner>/<repo> in its own org.
- When forge.ci.enable is set: an actions/checkout mirror is auto-appended +
  forgejo DEFAULT_ACTIONS_URL is pointed at this instance, so CI
  'uses: actions/checkout@vN' resolves on loopback — immune to a host-resolver
  blip that previously reded every checkout (the seed/re-sync needs external
  DNS, but that's off the CI critical path).
- forgejo-seed-mirrors.service: host-side oneshot (the core admin token never
  enters a container), modelled on hive-ci-prefetch — waits <=60s for the core
  token, then idempotently ensures each dest org + creates the pull-mirror via
  the migrate API. partOf the forge container so it re-ensures on restart.
- assertions: dest must be <owner>/<repo>; mirror orgs can't shadow the
  c0re-managed namespaces (config/shared/agents/core) so the seed never races
  hive-c0re's own provisioning.

Supersedes #2074 (the raw-clone stopgap) as the durable #2072 fix.
2026-06-29 00:26:41 +02:00
atlas
b0d099274e wip(#2072): forge.mirrors option + DEFAULT_ACTIONS_URL=self when CI on
General-purpose Forgejo pull-mirror config (services.hyperhive.forge.mirrors:
list of {upstream, dest}). When CI is enabled, auto-append an actions/checkout
mirror + point forgejo DEFAULT_ACTIONS_URL at this instance so CI's
actions/checkout@vN resolves on loopback (immune to host-resolver blips, #2072).

Seed oneshot (creates the dest orgs + pull-mirrors via the migrate API) is the
next commit.
2026-06-29 00:26:41 +02:00
2 changed files with 203 additions and 0 deletions

View file

@ -913,6 +913,98 @@ async fn ensure_org(name: &str, admin_token: &str) -> Result<()> {
}
}
/// One operator-declared pull-mirror, forwarded from the nix
/// `services.hyperhive.forge.mirrors` option as JSON in
/// `HYPERHIVE_FORGE_MIRRORS`.
#[derive(serde::Deserialize)]
struct Mirror {
/// Upstream clone URL to mirror from (e.g. `https://github.com/actions/checkout`).
upstream: String,
/// Local `<owner>/<repo>` the mirror is created at.
dest: String,
}
/// Ensure each `HYPERHIVE_FORGE_MIRRORS` entry exists as a real Forgejo
/// pull-mirror. The env carries the JSON-encoded nix `forge.mirrors` list
/// (plus the CI-auto `actions/checkout` entry). Absent/empty env = no-op.
/// Per-mirror failures warn and continue — never abort the startup sweep.
async fn ensure_mirrors(admin_token: &str) {
let raw = match std::env::var("HYPERHIVE_FORGE_MIRRORS") {
Ok(s) if !s.trim().is_empty() => s,
_ => return,
};
let mirrors: Vec<Mirror> = match serde_json::from_str(&raw) {
Ok(m) => m,
Err(e) => {
tracing::warn!(error = ?e, "forge: HYPERHIVE_FORGE_MIRRORS is not valid JSON; skipping mirror seed");
return;
}
};
for m in mirrors {
let Some((owner, repo)) = m.dest.split_once('/') else {
tracing::warn!(dest = %m.dest, "forge: mirror dest is not <owner>/<repo>; skipping");
continue;
};
// Create the dest org first (idempotent); the mirror can't land
// without its owner existing.
if let Err(e) = ensure_org(owner, admin_token).await {
tracing::warn!(%owner, error = ?e, "forge: ensure_org for mirror failed");
continue;
}
if let Err(e) = ensure_mirror_repo(&m.upstream, owner, repo, admin_token).await {
tracing::warn!(dest = %m.dest, error = ?e, "forge: ensure_mirror_repo failed");
}
}
}
/// Create `owner/repo` as a pull-mirror of `upstream` via the migrate API.
/// Idempotent: a cheap existence check skips an already-seeded mirror (it
/// persists in the non-ephemeral forge state across reboots); a 409 (a race
/// between that check and the POST) is also treated as success.
async fn ensure_mirror_repo(
upstream: &str,
owner: &str,
repo: &str,
admin_token: &str,
) -> Result<()> {
let get_url = format!("{FORGE_HTTP}/api/v1/repos/{owner}/{repo}");
let (status, _) = forge_http(reqwest::Method::GET, &get_url, admin_token, "").await?;
if status.is_success() {
tracing::debug!(%owner, %repo, "forge: pull-mirror already present");
return Ok(());
}
// serde_json::json! → the upstream URL is escaped safely (no string
// interpolation into the JSON body).
let body = serde_json::json!({
"clone_addr": upstream,
"repo_owner": owner,
"repo_name": repo,
"mirror": true,
"service": "git",
"private": false,
})
.to_string();
let url = format!("{FORGE_HTTP}/api/v1/repos/migrate");
let (status, text) = forge_http(reqwest::Method::POST, &url, admin_token, &body).await?;
match status.as_u16() {
201 => {
tracing::info!(%owner, %repo, %upstream, "forge: created pull-mirror");
Ok(())
}
// 409 = a race created it between our GET check and here (the GET
// is the real idempotency guard). NOT 422: for the migrate endpoint
// 422 is a validation error (bad clone_addr / service), so it must
// surface via the bail arm, not be swallowed as "already exists".
409 => {
tracing::debug!(%owner, %repo, "forge: pull-mirror already exists (race)");
Ok(())
}
other => {
anyhow::bail!("POST /api/v1/repos/migrate {owner}/{repo} returned HTTP {other}: {text}")
}
}
}
/// Whether `ns` is a hive-managed Forgejo namespace that agent-initiated
/// repo creation must never target — `internal` (operator-curated
/// shared content) + `agent-configs` / `core` (hive-c0re-internal). The
@ -1141,6 +1233,11 @@ pub async fn ensure_all() {
tracing::warn!(%org, error = ?e, "forge: ensure_org failed");
}
}
// Seed the operator-declared pull-mirrors (nix `forge.mirrors` +
// the CI-auto `actions/checkout`, forwarded via the
// `HYPERHIVE_FORGE_MIRRORS` env). Each ensures its own dest org, so
// this is independent of the SEEDED_ORGS loop above.
ensure_mirrors(token).await;
// Provision the operator merge-gate team (empty) inside the agents
// org so branch protection can reference it before anyone joins
//. The operator adds herself as a member out-of-band.

View file

@ -34,6 +34,25 @@ let
else
"http://${cfg.domain}:${toString cfg.httpPort}/";
effectiveRootUrl = if cfg.rootUrl != null then cfg.rootUrl else defaultRootUrl;
# When CI is enabled, the runner needs `actions/checkout` resolvable
# without external DNS (hive-ci shares the host netns, so a host-resolver
# blip otherwise reds every `actions/checkout@vN` fetch from
# data.forgejo.org). Auto-append a pull-mirror of it and point
# forgejo's DEFAULT_ACTIONS_URL at this instance so `uses:` resolves local.
ciEnabled = config.services.hyperhive.forge.ci.enable;
actionCheckoutMirror = {
upstream = "https://github.com/actions/checkout";
dest = "actions/checkout";
};
# Auto-append the actions/checkout mirror only when CI is on AND the
# operator hasn't already declared that dest themselves (else CI-on +
# an explicit `actions/checkout` entry would duplicate it).
effectiveMirrors =
cfg.mirrors
++ lib.optional (
ciEnabled && !(lib.any (m: m.dest == actionCheckoutMirror.dest) cfg.mirrors)
) actionCheckoutMirror;
in
{
# Private Forgejo in a `hive-forge` nixos-container, shared host
@ -177,6 +196,49 @@ in
config before rebuilding.
'';
};
mirrors = lib.mkOption {
type = lib.types.listOf (
lib.types.submodule {
options = {
upstream = lib.mkOption {
type = lib.types.str;
example = "https://github.com/actions/checkout";
description = "Upstream clone URL to mirror from.";
};
dest = lib.mkOption {
type = lib.types.str;
example = "actions/checkout";
description = ''
Local `<owner>/<repo>` the pull-mirror is created at. The
`<owner>` org is auto-created if missing. Keep mirror dests
in their own orgs (e.g. `actions/*`) separate from the
hive-c0re-managed namespaces (config/shared/agents/core) so
the seed never collides with core's own provisioning.
'';
};
};
}
);
default = [ ];
example = lib.literalExpression ''
[ { upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; } ]
'';
description = ''
General-purpose Forgejo **pull-mirrors** to auto-seed on the local
forge. Each entry is created as a real Forgejo pull-mirror (it
re-syncs from `upstream` out-of-band), not a one-off pushed clone
so a host-resolver blip leaves a *stale* mirror, never a hard
failure on whatever reads it.
When `services.hyperhive.forge.ci.enable` is set, an
`actions/checkout` mirror is auto-appended to this list and
forgejo's `DEFAULT_ACTIONS_URL` is pointed at this instance, so CI
`uses: actions/checkout@vN` steps resolve entirely on loopback with
no external DNS on the critical path (the seed/re-sync needs
external DNS, but that's off the CI path).
'';
};
};
config = lib.mkIf config.services.hyperhive.enable {
@ -207,6 +269,35 @@ in
hostname like "forge.example.com" or "git.internal".
'';
}
{
# Each mirror dest must be exactly `<owner>/<repo>` — the seed
# splits on the single slash to create the org + repo.
assertion = lib.all (m: lib.length (lib.splitString "/" m.dest) == 2) effectiveMirrors;
message = ''
Every services.hyperhive.forge.mirrors[].dest must be exactly
"<owner>/<repo>" (one slash). Got: ${lib.concatMapStringsSep ", " (m: m.dest) effectiveMirrors}
'';
}
{
# Keep mirror orgs out of the hive-c0re-managed namespaces
# (config/shared/agents/core) so the seed never races / collides
# with hive-c0re's own startup provisioning of those orgs.
assertion = lib.all (
m:
!(lib.elem (builtins.elemAt (lib.splitString "/" m.dest) 0) [
"config"
"shared"
"agents"
"core"
])
) effectiveMirrors;
message = ''
services.hyperhive.forge.mirrors[].dest must not place a mirror
in a hive-c0re-managed org (config / shared / agents / core)
those are provisioned by hive-c0re and a mirror there would
collide. Use a dedicated org (e.g. "actions/checkout").
'';
}
];
containers.hive-forge = {
@ -302,6 +393,13 @@ in
# of token scopes. Required by `hive-ci-register.service`
# in the hive-ci container.
actions.ENABLED = true;
# When CI is enabled, resolve `uses: <org>/<action>@vN` from
# THIS instance (the seeded `actions/checkout` pull-mirror)
# instead of the upstream default `data.forgejo.org` — keeps
# the checkout step on loopback, immune to a host-resolver
# blip. `self` = forgejo expands actions against its
# own ROOT_URL.
actions.DEFAULT_ACTIONS_URL = lib.mkIf ciEnabled "self";
# F3 (federation) computes its data dir relative to the
# forgejo binary, which lands in the read-only nix
# store and crashes anything that touches the F3
@ -412,5 +510,13 @@ in
cfg.sshPort
];
};
# Forward the declared pull-mirrors to hive-c0re, which seeds them in
# its forge provisioning sweep (`forge.rs::ensure_mirrors`, alongside
# the SEEDED_ORGS ensure). c0re already holds the core admin token and
# ensures the orgs there, so the seeding lives in one place rather than
# a parallel host-side unit. JSON-encoded list of { upstream, dest };
# `[]` when nothing to seed (c0re no-ops).
systemd.services.hive-c0re.environment.HYPERHIVE_FORGE_MIRRORS = builtins.toJSON effectiveMirrors;
};
}