vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-controller gains forge access when the token arrives, without a restart

Browse source 
A controller that started before the forge's host had published its token
ran without forge access until someone restarted it: on a split-host first
deploy that is a manual step. Now the forge client lives in a slot that is
filled once (`Late<T>`, an `Arc<OnceLock<T>>`):

- `forge::enable` tries the store once at start. On failure it logs once and
  re-reads in the background with backoff (30s doubling to a 5-minute cap,
  no log per retry), fills the slot and logs once when the token is there.
- The forge passes (#5014's per-hive token mint among them), the config-PR
  poll and the webhook registration start from that moment, through
  `start_forge`'s callback.
- Job nodes and HTTP handlers read the slot when they run, so they answer
  "no forge" before and work after.

A token replaced in the store after the slot is filled still reaches the
controller with its next restart. The round-1 change that shared
`queue_identity::fetch_with_retry` is reverted; nothing else uses it.

Refs #4998
This commit is contained in:
atlas 2026-10-08 13:40:24 +02:00
commit c45f7a5330
11 changed files with 243 additions and 135 deletions
Download patch file Download diff file Expand all files Collapse all files

185
swarm-controller/src/forge.rs
View file

@ -6,13 +6,17 @@
//!
//! The forge's URL comes from `swarm-controller.nix`'s `forgeEnv`
//! (`SWARM_CONTROLLER_FORGE_URL`). The token comes from the swarm's secret
//! store, at `swarm_secret_client::forge::controller_token_path`, read once at
//! start under this daemon's own certificate and held only in memory. The
//! forge mints it inside its container (`hive-forge/default.nix`'s
//! store, at `swarm_secret_client::forge::controller_token_path`, read under
//! this daemon's own certificate and held only in memory. The forge mints it
//! inside its container (`hive-forge/default.nix`'s
//! `forgejo-swarm-controller-account`) and `swarm-bao-forge-publisher`, on the
//! forge's host, puts it there. An unset URL means [`Client::from_env`]
//! returns `None` — every caller treats that as "no forge access here" and
//! continues, the same graceful-absence shape the queue coordinates use.
//! forge's host, puts it there.
//!
//! [`enable`] fills the daemon's one client slot: at start when the token is
//! there, later from a background re-read when it is not. Every caller reads
//! the slot and treats an empty one as "no forge access here" — the same
//! graceful-absence shape the queue coordinates use — so the daemon serves
//! from the start and gains forge access without a restart.
use anyhow::{Context, Result};
use forgejo_api::structs::{
@ -28,6 +32,7 @@ use futures_util::{StreamExt as _, TryStreamExt as _};
use reqwest::StatusCode;
use serde::{Deserialize, Serialize};
use std::collections::{BTreeMap, HashMap, HashSet};
use std::sync::{Arc, OnceLock};
use utoipa::ToSchema;
use crate::webhook::DeliveryKind;
@ -130,7 +135,7 @@ const KNOWLEDGE_REPO: &str = "knowledge";
const SWARM_CONTROLLER_FORGE_USER: &str = "swarm-controller";
/// The env var `swarm-controller.nix`'s `forgeEnv` sets. Named here once so
/// [`Client::from_env`] and the nix module can't drift silently.
/// [`url_from_env`] and the nix module can't drift silently.
const URL_ENV: &str = "SWARM_CONTROLLER_FORGE_URL";
/// One login and one read. An empty value is refused like an absent one: it
@ -163,34 +168,104 @@ pub struct Client {
url: url::Url,
}
/// The forge's URL. `Ok(None)` (not an error) when `SWARM_CONTROLLER_FORGE_URL`
/// is unset — no forge is configured for this daemon.
///
/// # Errors
/// A URL that does not parse. Retrying cannot fix that, so it is reported once
/// rather than waited out.
pub fn url_from_env() -> Result<Option<url::Url>> {
let Some(url) = std::env::var(URL_ENV).ok() else {
return Ok(None);
};
url::Url::parse(&url)
.with_context(|| format!("parse {URL_ENV} ({url}) as a URL"))
.map(Some)
}
/// How long the background re-read waits before its `n`-th attempt: 30s,
/// doubling, capped at 5 minutes.
fn background_delay(n: u32) -> std::time::Duration {
const FIRST_S: u64 = 30;
const CAP_S: u64 = 300;
std::time::Duration::from_secs(
FIRST_S
.saturating_mul(1u64.checked_shl(n).unwrap_or(u64::MAX))
.min(CAP_S),
)
}
/// Fill `slot` with the first value `attempt` yields, sleeping
/// [`background_delay`] before each attempt. Never gives up, and logs no
/// failure: the caller logs the first one and the recovery.
async fn fill_when_ready<T, F, Fut>(slot: &OnceLock<T>, mut attempt: F) -> &T
where
F: FnMut() -> Fut,
Fut: std::future::Future<Output = Result<T>>,
{
let mut n = 0;
loop {
tokio::time::sleep(background_delay(n)).await;
match attempt().await {
Ok(value) => return slot.get_or_init(|| value),
Err(e) => tracing::debug!(
error = %format!("{e:#}"),
"the controller's forge token is still unavailable"
),
}
n = n.saturating_add(1);
}
}
/// Put a client for `url` into `slot` and hand it to `on_ready`: at once when
/// the store has the token, otherwise from a background task that re-reads it
/// until it does. Either way the daemon starts, and every forge-backed path
/// reads `slot`, so it answers "no forge" until the token lands and works
/// from then on.
///
/// A token replaced in the store after the slot is filled reaches this daemon
/// with its next restart.
pub async fn enable(
url: url::Url,
slot: Arc<OnceLock<Arc<Client>>>,
on_ready: impl FnOnce(Arc<Client>) + Send + 'static,
) {
match Client::connect(&url).await {
Ok(client) => on_ready(Arc::clone(slot.get_or_init(|| Arc::new(client)))),
Err(e) => {
tracing::warn!(
error = %format!("{e:#}"),
"no forge access yet; re-reading the forge token from the swarm secret store in \
the background"
);
tokio::spawn(async move {
let client = Arc::clone(
fill_when_ready(&slot, || async {
Client::connect(&url).await.map(Arc::new)
})
.await,
);
tracing::info!("the forge token is in the swarm secret store; forge access is on");
on_ready(client);
});
}
}
}
impl Client {
/// Build from env and the secret store. `Ok(None)` (not an error) when
/// `SWARM_CONTROLLER_FORGE_URL` is unset — no forge is configured for this
/// daemon (see the module doc comment).
/// One read of the token from the secret store, and a client for `url`
/// holding it.
///
/// # Errors
/// An unparseable URL, or a token the store still would not give up after
/// `queue_identity::RETRY_DELAYS_S`: the store refusing the login, or
/// nothing at the path because the forge's host has not published it.
pub async fn from_env() -> Result<Option<Self>> {
let Some(url) = std::env::var(URL_ENV).ok() else {
return Ok(None);
};
let parsed_url =
url::Url::parse(&url).with_context(|| format!("parse {URL_ENV} ({url}) as a URL"))?;
let delays = crate::queue_identity::RETRY_DELAYS_S.map(std::time::Duration::from_secs);
let token = crate::queue_identity::fetch_with_retry(
&delays,
"the controller's forge token",
fetch_token,
)
.await?;
let api = Forgejo::new(Auth::Token(&token), parsed_url.clone())
.context("build forgejo client")?;
Ok(Some(Self {
/// The store refusing the login, nothing at the path because the forge's
/// host has not published it yet, or the client failing to build.
async fn connect(url: &url::Url) -> Result<Self> {
let token = fetch_token().await?;
let api = Forgejo::new(Auth::Token(&token), url.clone()).context("build forgejo client")?;
Ok(Self {
api,
url: parsed_url,
}))
url: url.clone(),
})
}
/// Creation options for an empty repo defaulting to `main`.
@ -1306,33 +1381,59 @@ mod tests {
// the same process-global env var would race each other. A single
// test's body is one thread, so sequential set/clear within it is
// race-free without needing a cross-test lock.
#[tokio::test]
async fn from_env_is_none_without_a_url_and_refuses_a_bad_one_before_the_store() {
#[test]
fn url_from_env_is_none_without_a_url_and_refuses_a_bad_one() {
// SAFETY: test-only mutation of the env var this module itself
// owns (`URL_ENV`), scoped to this one test function so there's no
// cross-test race (see comment above).
unsafe {
std::env::remove_var(URL_ENV);
}
assert!(Client::from_env().await.unwrap().is_none());
assert_eq!(url_from_env().unwrap(), None);
unsafe {
std::env::set_var(URL_ENV, "not a url");
}
// Not `.unwrap_err()`: that needs `Client: Debug` for its panic
// message, and the wrapped `forgejo_api::Forgejo` doesn't derive
// it. Returning at once, with no `BAO_*` set, is what shows the URL
// is checked before the store is asked.
match Client::from_env().await {
Err(e) => assert!(e.to_string().contains(URL_ENV), "{e:#}"),
Ok(_) => panic!("expected an error for an unparseable forge URL"),
}
let err = url_from_env().expect_err("an unparseable forge URL");
assert!(err.to_string().contains(URL_ENV), "{err:#}");
unsafe {
std::env::remove_var(URL_ENV);
}
}
#[test]
fn the_background_re_read_backs_off_from_30s_to_a_5_minute_cap() {
let s = |n| background_delay(n).as_secs();
assert_eq!([s(0), s(1), s(2), s(3), s(4)], [30, 60, 120, 240, 300]);
assert_eq!(s(40), 300);
assert_eq!(s(u32::MAX), 300);
}
// Paused time: each sleep completes at once and advances the clock, so the
// elapsed time is the backoff schedule itself.
#[tokio::test(start_paused = true)]
async fn a_token_absent_at_start_fills_the_slot_once_it_arrives() {
let slot = OnceLock::new();
let start = tokio::time::Instant::now();
let mut calls = 0;
let got = fill_when_ready(&slot, || {
calls += 1;
let n = calls;
async move {
if n < 3 {
anyhow::bail!("nothing at the controller's forge token path")
}
Ok("client")
}
})
.await;
assert_eq!(*got, "client");
assert_eq!(slot.get(), Some(&"client"));
assert_eq!(calls, 3);
assert_eq!(start.elapsed().as_secs(), 30 + 60 + 120);
}
#[tokio::test]
async fn page_search_results_pages_past_a_full_first_page() {
#[derive(Clone)]