vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-controller reads its forge token from the store, not a host file

Browse source 
The controller's forge token never entered bao: the forge minted it in its
container, hive-forge-swarm-controller-token copied it to a host file, and the
controller LoadCredential'ed that file. forgeTokenFile defaulted to that path
only where the forge ran, so a controller on any other host had no forge
access unless an operator copied the file by hand.

Now the forge's host publishes it and the controller reads it:

- swarm-bao-forge-publisher (replaces hive-forge-swarm-controller-token)
  waits for the in-container token, logs in with its own forge-publisher leaf
  and writes it to swarm/controller/swarm-controller/forge/token. It runs on
  every forge container start, the old unit's trigger.
- New principal swarm-forge-publisher: leaf minted by glue-bao-tls.nix,
  policy create/update on that one path, written by
  swarm-bao-forge-publisher-policy, CN reserved as a hive name.
- The controller policy gains read on that path. swarm-controller reads it
  at start under its own certificate with the OIDC secret's retry, and runs
  without forge access if it still cannot.
- forgeTokenFile, its LoadCredential and the forgeEnv gate are gone;
  SWARM_CONTROLLER_FORGE_URL is always set. hostSwarmControllerTokenFile is
  removed, and a tmpfiles rule deletes the old host copy on the forge host.

The in-container file stays: the mint unit's only-once guard reads it, and
it is the publisher's source.

Refs #4998
This commit is contained in:
atlas 2026-10-08 13:23:17 +02:00
commit 121c4678aa
18 changed files with 580 additions and 277 deletions
Download patch file Download diff file Expand all files Collapse all files

123
swarm-controller/src/forge.rs
View file

@ -4,16 +4,15 @@
//! calls, is premature plumbing; factor out a shared crate later if the
//! duplication actually starts to hurt).
//!
//! Reads its identity from the credentials `swarm-controller.nix`'s
//! `forgeEnv` wires in (`SWARM_CONTROLLER_FORGE_URL` +
//! `SWARM_CONTROLLER_FORGE_TOKEN_FILE`, the latter pointing at a
//! systemd `LoadCredential`-delivered file) — see that module and
//! `hive-forge/default.nix`'s `forgejo-swarm-controller-account` /
//! `hive-forge-swarm-controller-token` units for how the token itself
//! gets minted and delivered. Absent env (forge not configured on this
//! host) means [`Client::from_env`] returns `None` — every caller
//! treats that as "no forge access here" and continues, the same
//! graceful-absence shape the queue coordinates already use.
//! The forge's URL comes from `swarm-controller.nix`'s `forgeEnv`
//! (`SWARM_CONTROLLER_FORGE_URL`). The token comes from the swarm's secret
//! store, at `swarm_secret_client::forge::controller_token_path`, read once at
//! start under this daemon's own certificate and held only in memory. The
//! forge mints it inside its container (`hive-forge/default.nix`'s
//! `forgejo-swarm-controller-account`) and `swarm-bao-forge-publisher`, on the
//! forge's host, puts it there. An unset URL means [`Client::from_env`]
//! returns `None` — every caller treats that as "no forge access here" and
//! continues, the same graceful-absence shape the queue coordinates use.
use anyhow::{Context, Result};
use forgejo_api::structs::{
@ -130,12 +129,30 @@ const KNOWLEDGE_REPO: &str = "knowledge";
/// admin bypass, without adding a `git` shell-out to this daemon.
const SWARM_CONTROLLER_FORGE_USER: &str = "swarm-controller";
/// The env vars `swarm-controller.nix`'s `forgeEnv` sets. Named here
/// once so [`Client::from_env`] and the nix module can't drift silently
/// — a rename on one side without the other fails loudly (env var
/// absent) rather than silently losing forge access.
/// The env var `swarm-controller.nix`'s `forgeEnv` sets. Named here once so
/// [`Client::from_env`] and the nix module can't drift silently.
const URL_ENV: &str = "SWARM_CONTROLLER_FORGE_URL";
const TOKEN_FILE_ENV: &str = "SWARM_CONTROLLER_FORGE_TOKEN_FILE";
/// One login and one read. An empty value is refused like an absent one: it
/// would reach the forge as the token and come back as a 401 on the first
/// call, which names neither the store nor the publisher.
async fn fetch_token() -> Result<String> {
let path = swarm_secret_client::forge::controller_token_path()?;
let store = crate::store::connect()
.await
.context("logging in to the swarm secret store")?;
let stored: Option<swarm_secret_client::forge::ControllerToken> = store
.read_optional(&path)
.await
.with_context(|| format!("reading {path}"))?;
match stored {
Some(t) if !t.value.trim().is_empty() => Ok(t.value.trim().to_owned()),
_ => anyhow::bail!(
"nothing at {path}: swarm-bao-forge-publisher, on the host that runs the forge, has \
not published this daemon's forge token"
),
}
}
/// Typed forgejo client, built once at startup from env.
pub struct Client {
@ -147,38 +164,29 @@ pub struct Client {
}
impl Client {
/// Build from env. `Ok(None)` (not an error) when neither env var is
/// set — forge isn't configured on this host, which is a normal,
/// expected deployment shape (see the module doc comment), not a
/// failure. `Err` only for a genuine misconfiguration: one of the
/// two env vars present without the other, the token file unreadable,
/// or the URL/client failing to build — those ARE failures, because
/// they mean forge was *meant* to be configured here and isn't
/// usable, which should be loud rather than silently downgrading to
/// "no forge access."
pub fn from_env() -> Result<Option<Self>> {
let url = std::env::var(URL_ENV).ok();
let token_file = std::env::var(TOKEN_FILE_ENV).ok();
let (url, token_file) = match (url, token_file) {
(None, None) => return Ok(None),
(Some(url), Some(token_file)) => (url, token_file),
(Some(_), None) => {
anyhow::bail!("{URL_ENV} is set but {TOKEN_FILE_ENV} is not — partial forge config")
}
(None, Some(_)) => {
anyhow::bail!("{TOKEN_FILE_ENV} is set but {URL_ENV} is not — partial forge config")
}
/// Build from env and the secret store. `Ok(None)` (not an error) when
/// `SWARM_CONTROLLER_FORGE_URL` is unset — no forge is configured for this
/// daemon (see the module doc comment).
///
/// # Errors
/// An unparseable URL, or a token the store still would not give up after
/// `queue_identity::RETRY_DELAYS_S`: the store refusing the login, or
/// nothing at the path because the forge's host has not published it.
pub async fn from_env() -> Result<Option<Self>> {
let Some(url) = std::env::var(URL_ENV).ok() else {
return Ok(None);
};
let token = std::fs::read_to_string(&token_file)
.with_context(|| format!("read forge token from {token_file}"))?;
let token = token.trim();
if token.is_empty() {
anyhow::bail!("forge token file {token_file} is empty");
}
let parsed_url =
url::Url::parse(&url).with_context(|| format!("parse {URL_ENV} ({url}) as a URL"))?;
let api =
Forgejo::new(Auth::Token(token), parsed_url.clone()).context("build forgejo client")?;
let delays = crate::queue_identity::RETRY_DELAYS_S.map(std::time::Duration::from_secs);
let token = crate::queue_identity::fetch_with_retry(
&delays,
"the controller's forge token",
fetch_token,
)
.await?;
let api = Forgejo::new(Auth::Token(&token), parsed_url.clone())
.context("build forgejo client")?;
Ok(Some(Self {
api,
url: parsed_url,
@ -1295,30 +1303,29 @@ mod tests {
// Both scenarios live in ONE test, not two, deliberately: `cargo
// test` runs tests in parallel by default, and two tests mutating
// the same process-global env vars would race each other. A single
// the same process-global env var would race each other. A single
// test's body is one thread, so sequential set/clear within it is
// race-free without needing a cross-test lock.
#[test]
fn from_env_reads_both_vars_or_neither() {
// SAFETY: test-only mutation of env vars this module itself
// owns (`URL_ENV`/`TOKEN_FILE_ENV`), scoped to this one test
// function so there's no cross-test race (see comment above).
#[tokio::test]
async fn from_env_is_none_without_a_url_and_refuses_a_bad_one_before_the_store() {
// SAFETY: test-only mutation of the env var this module itself
// owns (`URL_ENV`), scoped to this one test function so there's no
// cross-test race (see comment above).
unsafe {
std::env::remove_var(URL_ENV);
std::env::remove_var(TOKEN_FILE_ENV);
}
assert!(Client::from_env().unwrap().is_none());
assert!(Client::from_env().await.unwrap().is_none());
unsafe {
std::env::set_var(URL_ENV, "https://forge.example.invalid");
std::env::set_var(URL_ENV, "not a url");
}
// Not `.unwrap_err()`: that needs `Client: Debug` for its panic
// message, and the wrapped `forgejo_api::Forgejo` doesn't derive
// it. Match directly instead of adding a `Debug` impl nothing
// else needs.
match Client::from_env() {
Err(e) => assert!(e.to_string().contains(TOKEN_FILE_ENV)),
Ok(_) => panic!("expected an error for a partial forge config"),
// it. Returning at once, with no `BAO_*` set, is what shows the URL
// is checked before the store is asked.
match Client::from_env().await {
Err(e) => assert!(e.to_string().contains(URL_ENV), "{e:#}"),
Ok(_) => panic!("expected an error for an unparseable forge URL"),
}
unsafe {

43
swarm-controller/src/main.rs
View file

@ -313,8 +313,8 @@ async fn run_swarm_node(
},
SwarmNodeKind::CreateRepo { agent } => match deps.forge {
None => Outcome::Failed(
"no forge configured on this host (SWARM_CONTROLLER_FORGE_URL / \
SWARM_CONTROLLER_FORGE_TOKEN_FILE unset)"
"no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
not in the swarm secret store)"
.to_owned(),
),
Some(client) => match client.create_repo(&agent).await {
@ -327,8 +327,8 @@ async fn run_swarm_node(
},
SwarmNodeKind::CreateForgeUser { agent } => match deps.forge {
None => Outcome::Failed(
"no forge configured on this host (SWARM_CONTROLLER_FORGE_URL / \
SWARM_CONTROLLER_FORGE_TOKEN_FILE unset)"
"no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
not in the swarm secret store)"
.to_owned(),
),
Some(client) => match client.ensure_agent_user(&agent).await {
@ -338,8 +338,8 @@ async fn run_swarm_node(
},
SwarmNodeKind::AddRepoMember { agent } => match deps.forge {
None => Outcome::Failed(
"no forge configured on this host (SWARM_CONTROLLER_FORGE_URL / \
SWARM_CONTROLLER_FORGE_TOKEN_FILE unset)"
"no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
not in the swarm secret store)"
.to_owned(),
),
Some(client) => match client.add_repo_member(&agent, &agent).await {
@ -350,8 +350,8 @@ async fn run_swarm_node(
SwarmNodeKind::GrantDocsRead { agent } => grant_docs_read(deps.forge, &agent).await,
SwarmNodeKind::InitAgentConfigRepo { agent } => match deps.forge {
None => Outcome::Failed(
"no forge configured on this host (SWARM_CONTROLLER_FORGE_URL / \
SWARM_CONTROLLER_FORGE_TOKEN_FILE unset)"
"no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
not in the swarm secret store)"
.to_owned(),
),
Some(client) => match client.seed_agent_config(&agent, &agent).await {
@ -411,8 +411,8 @@ async fn mint_forge_token(
let Some(client) = forge else {
return Outcome::Failed(
"no forge configured on this host (SWARM_CONTROLLER_FORGE_URL / \
SWARM_CONTROLLER_FORGE_TOKEN_FILE unset), so no agent forge token can be minted"
"no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
not in the swarm secret store), so no agent forge token can be minted"
.to_owned(),
);
};
@ -431,8 +431,8 @@ async fn grant_docs_read(
let Some(client) = forge else {
return Outcome::Failed(
"no forge configured on this host (SWARM_CONTROLLER_FORGE_URL / \
SWARM_CONTROLLER_FORGE_TOKEN_FILE unset), so no docs grant can be made"
"no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
not in the swarm secret store), so no docs grant can be made"
.to_owned(),
);
};
@ -461,8 +461,8 @@ async fn mint_hive_forge_token(
let Some(client) = forge else {
return Outcome::Failed(
"no forge configured on this host (SWARM_CONTROLLER_FORGE_URL / \
SWARM_CONTROLLER_FORGE_TOKEN_FILE unset), so no hive forge token can be minted"
"no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
not in the swarm secret store), so no hive forge token can be minted"
.to_owned(),
);
};
@ -3017,13 +3017,14 @@ async fn main() -> Result<()> {
None
}
};
// Same shape again: a controller with no forge configured still serves
// Same shape again: a controller with no forge access still serves
// everything else, and the forge-shaped node kinds give an honest
// per-job failure rather than this fn refusing to start.
let forge_client = match forge::Client::from_env() {
// per-job failure rather than this fn refusing to start. A token missing
// from the store lands here too, after `from_env`'s retries.
let forge_client = match forge::Client::from_env().await {
Ok(client) => client.map(Arc::new),
Err(e) => {
tracing::warn!(error = %format!("{e:#}"), "forge misconfigured; repo provisioning is off");
tracing::warn!(error = %format!("{e:#}"), "no forge access; repo provisioning is off");
None
}
};
@ -3250,7 +3251,7 @@ mod tests {
/// interesting behavior is in `run_swarm_node`'s executor arm, which
/// this exercises directly).
///
/// Deliberately offline: with both forge env vars unset,
/// Deliberately offline: with the forge URL unset,
/// `forge::Client::from_env` returns `Ok(None)` (see that module's doc
/// comment), so this exercises the whole claim → run → complete path
/// through `hive_jobq::scheduler::Scheduler` without a real forge
@ -3259,7 +3260,7 @@ mod tests {
/// actual forge instance and isn't something a unit test in this crate
/// can reach.
///
/// SAFETY: single-threaded mutation of the two `forge` env vars this
/// SAFETY: single-threaded mutation of the `forge` env var this
/// test itself owns, restored before returning — no other test in this
/// crate reads them.
/// One roster, one agent name, two hives — the accepted one and a
@ -4177,7 +4178,6 @@ mod tests {
async fn create_repo_node_runs_end_to_end_and_fails_without_forge_configured() {
unsafe {
std::env::remove_var("SWARM_CONTROLLER_FORGE_URL");
std::env::remove_var("SWARM_CONTROLLER_FORGE_TOKEN_FILE");
}
let mut sched = hive_jobq::scheduler::Scheduler::new(
@ -4231,7 +4231,6 @@ mod tests {
async fn create_forge_user_node_runs_end_to_end_and_fails_without_forge_configured() {
unsafe {
std::env::remove_var("SWARM_CONTROLLER_FORGE_URL");
std::env::remove_var("SWARM_CONTROLLER_FORGE_TOKEN_FILE");
}
let mut sched = hive_jobq::scheduler::Scheduler::new(

23
swarm-controller/src/queue_identity.rs
View file

@ -25,7 +25,7 @@ const PREFIX: &str = "SWARM_CONTROLLER";
/// Seconds to wait after each failed fetch before the next one. About a minute
/// in all, which rides out a store restarting alongside this daemon. A longer
/// outage fails the start, and the unit's `Restart=` tries again from the top.
const RETRY_DELAYS_S: [u64; 6] = [1, 2, 4, 8, 16, 30];
pub(crate) const RETRY_DELAYS_S: [u64; 6] = [1, 2, 4, 8, 16, 30];
static IDENTITY: OnceLock<Option<QueueConfig>> = OnceLock::new();
@ -61,7 +61,9 @@ pub async fn init() -> Result<()> {
None => None,
Some(coordinates) => {
let delays = RETRY_DELAYS_S.map(Duration::from_secs);
Some(coordinates.with_secret(fetch_with_retry(&delays, fetch_secret).await?))
Some(coordinates.with_secret(
fetch_with_retry(&delays, "the queue client secret", fetch_secret).await?,
))
}
};
IDENTITY
@ -118,8 +120,13 @@ async fn fetch_secret() -> Result<String> {
}
/// Run `attempt`, waiting each of `delays` after a failure, then once more.
/// Answers with the first success, or the last attempt's error.
async fn fetch_with_retry<F, Fut>(delays: &[Duration], mut attempt: F) -> Result<String>
/// Answers with the first success, or the last attempt's error. `what` names the
/// secret in the log line and the error.
pub(crate) async fn fetch_with_retry<F, Fut>(
delays: &[Duration],
what: &str,
mut attempt: F,
) -> Result<String>
where
F: FnMut() -> Fut,
Fut: Future<Output = Result<String>>,
@ -131,7 +138,7 @@ where
tracing::warn!(
error = %format!("{e:#}"),
retry_in = ?delay,
"fetching the queue client secret from the swarm secret store failed"
"fetching {what} from the swarm secret store failed"
);
tokio::time::sleep(*delay).await;
}
@ -139,7 +146,7 @@ where
}
attempt().await.with_context(|| {
format!(
"fetching the queue client secret from the swarm secret store, after {} attempts",
"fetching {what} from the swarm secret store, after {} attempts",
delays.len() + 1
)
})
@ -217,7 +224,7 @@ mod tests {
#[tokio::test]
async fn a_store_that_comes_back_within_the_window_is_waited_for() {
let mut calls = 0;
let got = fetch_with_retry(&[Duration::ZERO; 3], || {
let got = fetch_with_retry(&[Duration::ZERO; 3], "a secret", || {
calls += 1;
let n = calls;
async move {
@ -236,7 +243,7 @@ mod tests {
#[tokio::test]
async fn a_store_that_stays_down_fails_after_one_attempt_per_delay_plus_one() {
let mut calls = 0;
let err = fetch_with_retry(&[Duration::ZERO; 3], || {
let err = fetch_with_retry(&[Duration::ZERO; 3], "a secret", || {
calls += 1;
async { bail!("store sealed") }
})