vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-controller: split main.rs into jobq, routes and startup modules

Browse source 
no behaviour change: a pure move of items and tests into new modules, plus
visibility edits (pub(crate)) and path updates in sibling modules that named
crate-root items. same routes, same OpenAPI, 561 tests before and after.

why: #5470, new features stop landing in one 6.4k-line file by default.

files outside swarm-controller/src: docs/agent-lifecycle/agent-roster.md,
swarm-ui JobsPage.tsx and AgentsPage.tsx, nix/host-modules/swarm-controller.nix
and swarm-grafana.nix hold comments that named main.rs for items that moved.
This commit is contained in:
atlas 2026-10-18 04:53:59 +02:00
commit 738dd1c95b
42 changed files with 6854 additions and 6680 deletions
Download patch file Download diff file Expand all files Collapse all files

2404
swarm-controller/src/routes/agents.rs Normal file
View file

File diff suppressed because it is too large Load diff

81
swarm-controller/src/routes/config_prs.rs Normal file
View file

@ -0,0 +1,81 @@
//! Agents' open config PRs.
use anyhow::Result;
use axum::{
Json,
extract::{Path, State},
};
use crate::forge;
use crate::routes::{AppState, StatusUnavailable};
/// `agent`'s open config PR, from [`ConfigPrCache`](crate::config_pr::ConfigPrCache) — kept
/// current by a webhook nudge (near-real-time) backstopped by a periodic
/// scan (worst case: up to one `config_pr::POLL_INTERVAL` stale if a
/// delivery was ever missed). Not a live forge read either way, so this
/// answers even when the forge itself is momentarily unreachable.
///
/// `200` with a `null` body means "no open PR" (or "no scan has completed
/// yet") — not distinguished, for the same reason
/// [`ConfigPrCache::get`](crate::config_pr::ConfigPrCache::get) doesn't: both read as "nothing to show"
/// to the swarm-ui config-PR panel this feeds, and a cache is a best-effort
/// read, not a source of truth callers should expect to disambiguate
/// against.
#[utoipa::path(
get,
path = "/api/agents/{name}/config-pr",
params(("name" = String, Path, description = "agent name")),
responses(
(status = 200, description = "the agent's open config PR, or null if none / not scanned yet", body = Option<forge::ConfigPrStatus>),
(status = 503, description = "no forge is configured on this host", body = String),
),
tag = "agents"
)]
pub(crate) async fn get_agent_config_pr(
State(state): State<AppState>,
Path(name): Path<String>,
) -> Result<Json<Option<forge::ConfigPrStatus>>, StatusUnavailable> {
let Some(cache) = state.config_prs.get() else {
return Err(StatusUnavailable(
"no forge is configured on this host".to_owned(),
));
};
Ok(Json(cache.get(&name)))
}
/// Every agent with an open config PR, in one response — the bulk
/// counterpart to [`get_agent_config_pr`]. swarm-ui's config-PR table needs
/// every agent's status to render, and fetching them one at a time doesn't
/// scale and isn't how the rest of swarm-ui's single-fetch pages (jobq,
/// hives status) work.
///
/// Deliberately **not** `/api/agents/config-prs`: agent names are
/// user-specified (any string `hive_types::Ident` accepts), so a literal
/// path segment sitting where a `{name}` capture could plausibly also want
/// to live is a real, not theoretical, clash risk the moment someone names
/// an agent `config-prs` — per mara's review call, closed off by
/// construction rather than relying on axum's static-route-priority
/// tiebreak to paper over it.
///
/// Only agents with a currently-open PR are present — same "absence is the
/// answer" shape [`get_agent_config_pr`]'s `null` uses, just at map-entry
/// granularity instead of per-request.
#[utoipa::path(
get,
path = "/api/config-prs",
responses(
(status = 200, description = "agent name -> open config PR, only agents with one present", body = std::collections::HashMap<String, forge::ConfigPrStatus>),
(status = 503, description = "no forge is configured on this host", body = String),
),
tag = "agents"
)]
pub(crate) async fn get_config_prs(
State(state): State<AppState>,
) -> Result<Json<std::collections::HashMap<String, forge::ConfigPrStatus>>, StatusUnavailable> {
let Some(cache) = state.config_prs.get() else {
return Err(StatusUnavailable(
"no forge is configured on this host".to_owned(),
));
};
Ok(Json(cache.snapshot()))
}

236
swarm-controller/src/routes/forge_admin.rs Normal file
View file

@ -0,0 +1,236 @@
//! Forge site admins and an agent's forge token, on request.
use anyhow::Result;
use axum::{
Json,
extract::{Path, State},
};
use swarm_controller_api::agents::MintAgentForgeTokenResponse;
use swarm_controller_api::forge::{ForgeAdminChange, MakeForgeAdminResponse};
use crate::forge;
use crate::jobq::queue_forge_token_mints;
use crate::routes::agents::in_roster;
use crate::routes::{AppState, error_problem};
/// Check an agent's forge token now, and mint one if it is missing or stale.
///
/// The periodic pass (`forge::agent_token::spawn`) does the same every five
/// minutes for every agent with a store identity; this is for an operator who
/// does not want to wait, or for an agent the pass skipped. Queues the same
/// node agent creation does. Idempotent: a current token is left alone.
#[utoipa::path(
post,
path = "/api/agents/{name}/forge-token",
params(("name" = String, Path, description = "agent name")),
responses(
(status = 200, description = "mint queued", body = MintAgentForgeTokenResponse),
(status = 400, description = "`name` is not a valid identifier (problem+json)", body = String),
(status = 500, description = "the job could not be queued (problem+json)", body = String),
),
tag = "agents"
)]
pub(crate) async fn mint_agent_forge_token(
State(state): State<AppState>,
Path(name): Path<String>,
) -> Result<Json<MintAgentForgeTokenResponse>, problem_details::ProblemDetails> {
let agent = hive_types::Ident::parse(&name)
.map_err(|reason| error_problem(axum::http::StatusCode::BAD_REQUEST, reason))?
.into_string();
let ids = queue_forge_token_mints(&state.jobq, vec![agent]).map_err(|e| {
error_problem(
axum::http::StatusCode::INTERNAL_SERVER_ERROR,
&e.to_string(),
)
})?;
let [id] = ids[..] else {
unreachable!("exactly one handle was asked for");
};
Ok(Json(MintAgentForgeTokenResponse { node_id: id.get() }))
}
/// Make an existing human forge account a site admin. Idempotent.
///
/// Never creates the account: a human's is made by their first SSO login to
/// the forge, and until then this answers 404. Refuses an agent's name: a
/// site admin can create repos whatever its `max_repo_creation`, and an
/// agent's account is locked to 0 so its token cannot create a repo and
/// self-merge in it.
#[utoipa::path(
post,
path = "/api/forge/users/{name}/admin",
params(("name" = String, Path, description = "forge username")),
responses(
(status = 200, description = "the account is a site admin", body = MakeForgeAdminResponse),
(status = 400, description = "`name` is not a valid identifier (problem+json)", body = String),
(status = 404, description = "no such account: the user has not logged in via SSO yet (problem+json)", body = String),
(status = 409, description = "`name` is an agent (problem+json)", body = String),
(status = 500, description = "the forge refused the read or the edit (problem+json)", body = String),
(status = 503, description = "no forge configured, or the agent roster could not be read (problem+json)", body = String),
),
tag = "forge"
)]
pub(crate) async fn make_forge_admin(
State(state): State<AppState>,
Path(name): Path<String>,
) -> Result<Json<MakeForgeAdminResponse>, problem_details::ProblemDetails> {
let name = hive_types::Ident::parse(&name)
.map_err(|reason| error_problem(axum::http::StatusCode::BAD_REQUEST, reason))?
.into_string();
let Some(forge) = state.forge.get().cloned() else {
return Err(error_problem(
axum::http::StatusCode::SERVICE_UNAVAILABLE,
"no forge is configured on this host",
));
};
let is_agent = in_roster(state.auth.as_deref(), &name).await;
admin_unless_agent(&name, is_agent, || async {
forge.make_site_admin(&name).await
})
.await
}
/// Run `promote` only if `name` is known not to be an agent, and answer with
/// what it found.
///
/// An unreadable roster refuses too: a site admin is the one thing an agent's
/// account must never become, so "could not tell" is not "no".
pub(crate) async fn admin_unless_agent<F, Fut>(
name: &str,
is_agent: Result<bool, String>,
promote: F,
) -> Result<Json<MakeForgeAdminResponse>, problem_details::ProblemDetails>
where
F: FnOnce() -> Fut,
Fut: std::future::Future<Output = Result<forge::site_admin::Plan>>,
{
match is_agent {
Ok(false) => {}
Ok(true) => {
return Err(error_problem(
axum::http::StatusCode::CONFLICT,
&format!("{name:?} is an agent; an agent's forge account is never a site admin"),
));
}
Err(e) => {
return Err(error_problem(
axum::http::StatusCode::SERVICE_UNAVAILABLE,
&format!("cannot tell whether {name:?} is an agent, so refusing: {e}"),
));
}
}
let plan = promote().await.map_err(|e| {
error_problem(
axum::http::StatusCode::INTERNAL_SERVER_ERROR,
&format!("{e:#}"),
)
})?;
let change = match plan {
forge::site_admin::Plan::NoSuchUser => {
return Err(error_problem(
axum::http::StatusCode::NOT_FOUND,
&format!(
"the forge has no user {name:?}: the user has not logged in via SSO yet. \
The forge creates the account on their first authelia login; run this \
again after it"
),
));
}
forge::site_admin::Plan::AlreadyAdmin => ForgeAdminChange::AlreadyAdmin,
forge::site_admin::Plan::Promote => ForgeAdminChange::Promoted,
};
Ok(Json(MakeForgeAdminResponse { change }))
}
#[cfg(test)]
mod tests {
use super::*;
/// `admin_unless_agent` with a `promote` that records whether it ran and
/// answers `plan`.
async fn forge_admin_route(
is_agent: Result<bool, String>,
plan: crate::forge::site_admin::Plan,
) -> (
Result<axum::Json<super::MakeForgeAdminResponse>, problem_details::ProblemDetails>,
bool,
) {
let ran = std::sync::atomic::AtomicBool::new(false);
let result = super::admin_unless_agent("atlas", is_agent, || async {
ran.store(true, std::sync::atomic::Ordering::SeqCst);
Ok(plan)
})
.await;
(result, ran.into_inner())
}
/// An agent's name never reaches the forge.
#[tokio::test]
async fn the_forge_admin_route_refuses_an_agent() {
let (result, ran) =
forge_admin_route(Ok(true), crate::forge::site_admin::Plan::Promote).await;
let Err(problem) = result else {
panic!("an agent's name must be refused");
};
assert!(!ran, "the account must not be touched");
assert_eq!(problem.status, Some(axum::http::StatusCode::CONFLICT));
}
/// A roster that can't be read can't vouch for the name, so the route
/// refuses rather than guessing "not an agent".
#[tokio::test]
async fn the_forge_admin_route_refuses_on_an_unreadable_roster() {
let (result, ran) = forge_admin_route(
Err("bridge down".to_owned()),
crate::forge::site_admin::Plan::Promote,
)
.await;
let Err(problem) = result else {
panic!("an unreadable roster must refuse");
};
assert!(!ran, "the account must not be touched");
assert_eq!(
problem.status,
Some(axum::http::StatusCode::SERVICE_UNAVAILABLE)
);
}
/// A missing account is a 404 that tells the operator what to wait for.
#[tokio::test]
async fn the_forge_admin_route_names_the_missing_sso_login() {
let (result, ran) =
forge_admin_route(Ok(false), crate::forge::site_admin::Plan::NoSuchUser).await;
let Err(problem) = result else {
panic!("a missing account must not report success");
};
assert!(ran);
assert_eq!(problem.status, Some(axum::http::StatusCode::NOT_FOUND));
let rendered = format!("{problem:?}");
assert!(
rendered.contains("has not logged in via SSO yet"),
"{rendered}"
);
}
#[tokio::test]
async fn the_forge_admin_route_promotes_a_non_agent() {
let (result, ran) =
forge_admin_route(Ok(false), crate::forge::site_admin::Plan::Promote).await;
let Ok(axum::Json(resp)) = result else {
panic!("a non-agent must be promoted");
};
assert!(ran);
assert_eq!(resp.change, super::ForgeAdminChange::Promoted);
}
/// Idempotent: a second run succeeds and says nothing changed.
#[tokio::test]
async fn the_forge_admin_route_succeeds_for_an_admin() {
let (result, _) =
forge_admin_route(Ok(false), crate::forge::site_admin::Plan::AlreadyAdmin).await;
let Ok(axum::Json(resp)) = result else {
panic!("an existing admin must succeed");
};
assert_eq!(resp.change, super::ForgeAdminChange::AlreadyAdmin);
}
}

126
swarm-controller/src/routes/hives.rs Normal file
View file

@ -0,0 +1,126 @@
//! The swarm's hive directory, service links and display name.
use anyhow::Result;
use axum::{Json, extract::State};
use serde::{Deserialize, Serialize};
use utoipa::ToSchema;
use crate::routes::{AppState, StatusUnavailable};
use crate::status;
/// One hive in the swarm's directory — the same `name`/`domain` pair
/// `services.hyperhive.swarm.hives` (nix/host-modules/swarm.nix) declares,
/// plus that hive's `swarm.dashboardDomains` entry, carried across
/// unchanged rather than reshaped, so this stays a direct mirror of the
/// nix source of truth instead of a second vocabulary for the same fields.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, ToSchema)]
pub(crate) struct HiveEntry {
pub(crate) name: String,
pub(crate) domain: String,
/// The host the hive's dashboard answers on.
pub(crate) dashboard: String,
}
/// The swarm's hive directory — every hive, including whichever one this
/// controller instance happens to run on (there is no "self" to exclude
/// at the swarm level, unlike `hive-c0re`'s peer list).
#[utoipa::path(
get,
path = "/api/hives",
responses((status = 200, description = "every hive in the swarm", body = Vec<HiveEntry>)),
tag = "hives"
)]
pub(crate) async fn get_hives(State(state): State<AppState>) -> Json<Vec<HiveEntry>> {
Json((*state.hives).clone())
}
/// One quick link to a swarm-wide service (authelia, matrix, forge, this
/// daemon's own swagger UI, …). Deliberately generic rather than named
/// fields per service: each service's own nix module contributes its own
/// entry to `services.hyperhive.swarm.controller.links` (same list-merge
/// idiom `services.hyperhive.gateway.localNames` already uses), so adding
/// a new one is a nix-only change — no new field here, no swarm-ui change.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, ToSchema)]
pub(crate) struct ServiceLink {
pub(crate) label: String,
/// Emoji or short glyph. Empty string, not `Option`, when a
/// contributing module has none — one fewer null-vs-absent case for
/// the frontend to handle, and every real contributor sets one today.
pub(crate) icon: String,
pub(crate) url: String,
}
/// Quick links to swarm-wide services (authelia, matrix, forge, this
/// daemon's own swagger UI, …), as contributed by each service's own nix
/// module. Empty when nothing was configured to contribute — a caller
/// renders 0 links the same way it renders any other count, no special
/// "not configured" case.
#[utoipa::path(
get,
path = "/api/links",
responses((status = 200, description = "swarm service quick links", body = Vec<ServiceLink>)),
tag = "links"
)]
pub(crate) async fn get_links(State(state): State<AppState>) -> Json<Vec<ServiceLink>> {
Json((*state.links).clone())
}
/// Body of `GET /api/swarm`.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, ToSchema)]
pub(crate) struct SwarmInfo {
/// `None` when the operator never set `services.hyperhive.swarm.name`
/// — a caller (swarm-ui's chrome) falls back to a generic label rather
/// than treating this as an error.
name: Option<String>,
}
/// The swarm's own display name, for UI chrome (page title, nav) that
/// wants to say which swarm it's showing — distinct from [`get_hives`],
/// which lists the *hives inside* the swarm, not the swarm itself.
#[utoipa::path(
get,
path = "/api/swarm",
responses((status = 200, description = "the swarm's display name, if the operator set one", body = SwarmInfo)),
tag = "hives"
)]
pub(crate) async fn get_swarm_info(State(state): State<AppState>) -> Json<SwarmInfo> {
Json(SwarmInfo {
name: state.swarm_name.as_deref().map(str::to_owned),
})
}
/// What each hive last said about itself, read from the swarm queue at
/// request time.
///
/// Every hive in the roster gets a row whether or not it has ever
/// reported — see the `status` module for why absence, not presence, is
/// the case this is built around.
#[utoipa::path(
get,
path = "/api/hives/status",
responses(
(status = 200, description = "a row per hive, freshness derived now", body = Vec<status::HiveStatus>),
(status = 503, description = "no swarm queue is configured here, or its store could not be read", body = String),
),
tag = "hives"
)]
pub(crate) async fn get_hives_status(
State(state): State<AppState>,
) -> Result<Json<Vec<status::HiveStatus>>, StatusUnavailable> {
let Some(reader) = state.status.as_ref() else {
return Err(StatusUnavailable(
"no swarm queue is configured on this host".to_owned(),
));
};
match reader
.view(&state.hives, std::time::SystemTime::now())
.await
{
Ok(rows) => Ok(Json(rows)),
Err(e) => {
let detail = format!("{e:#}");
tracing::warn!(error = %detail, "reading the swarm status bucket failed");
Err(StatusUnavailable(detail))
}
}
}

65
swarm-controller/src/routes/jobq_graph.rs Normal file
View file

@ -0,0 +1,65 @@
//! Read-only views of the swarm-level job graph.
use axum::{Json, extract::State};
use hive_jobq_wire::GraphWire as _;
use serde::Deserialize;
use crate::routes::AppState;
/// Query params for `GET /api/jobq/graph` — `?states=` narrows to root
/// groups in the named states, same shape `hive_jobq_wire::parse_states`
/// parses.
#[derive(Deserialize, utoipa::IntoParams)]
pub(crate) struct JobqGraphQuery {
states: Option<String>,
}
/// Every node of every root group in the swarm-level job graph. Nothing
/// filters "done and old" here — with zero nodes ever submitted there is
/// nothing to bound yet, so `graph.roots()` (everything) is the whole
/// roster passed to [`hive_jobq_wire::GraphWire::wire_snapshot`].
#[utoipa::path(
get,
path = "/api/jobq/graph",
params(JobqGraphQuery),
responses((status = 200, description = "every node of every root group, as generic \
`hive_jobq` graph nodes. `?states=` narrows to root groups in the named states.",
body = Vec<hive_jobq_wire::GraphNode>)),
tag = "jobq"
)]
pub(crate) async fn get_jobq_graph(
State(state): State<AppState>,
axum::extract::Query(q): axum::extract::Query<JobqGraphQuery>,
) -> Json<Vec<hive_jobq_wire::GraphNode>> {
let states = hive_jobq_wire::parse_states(q.states.as_deref());
let sched = state
.jobq
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let graph = sched.graph();
let roots: Vec<hive_jobq::NodeId> = graph.roots().map(|n| n.id).collect();
let nodes = graph.wire_snapshot(roots);
Json(hive_jobq_wire::filter_nodes_by_state(
nodes,
states.as_deref(),
))
}
/// Counts by lifecycle state over the same groups `/api/jobq/graph` serves.
#[utoipa::path(
get,
path = "/api/jobq/rollup",
responses((status = 200, description = "counts by lifecycle state", body = Vec<hive_jobq_wire::StateCount>)),
tag = "jobq"
)]
pub(crate) async fn get_jobq_rollup(
State(state): State<AppState>,
) -> Json<Vec<hive_jobq_wire::StateCount>> {
let sched = state
.jobq
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let graph = sched.graph();
let roots: Vec<hive_jobq::NodeId> = graph.roots().map(|n| n.id).collect();
Json(hive_jobq_wire::state_rollup(graph, roots))
}

365
swarm-controller/src/routes/mod.rs Normal file
View file

@ -0,0 +1,365 @@
//! The HTTP state every handler shares, the error shapes they answer with,
//! and the handlers that live outside a resource module of their own.
pub(crate) mod agents;
pub(crate) mod config_prs;
pub(crate) mod forge_admin;
pub(crate) mod hives;
pub(crate) mod jobq_graph;
use std::sync::{Arc, Mutex};
use crate::jobq::{Late, SwarmNodeKind, SwarmResourceKind};
use crate::routes::hives::{HiveEntry, ServiceLink};
use crate::{
agent_icon, agent_login, agent_status, agents_md, alerts, auth, builder, config_pr, forge,
inference, linked_accounts, review_classifier, status, wanted,
};
/// Liveness probe. Returns the build's version so an operator can tell
/// *which* controller answered without shelling onto the host.
#[utoipa::path(
get,
path = "/health",
responses((status = 200, description = "process is up, body is \"swarm-controller <version>\"", body = String)),
tag = "health"
)]
pub(crate) async fn health() -> &'static str {
concat!("swarm-controller ", env!("CARGO_PKG_VERSION"), "\n")
}
#[derive(Clone)]
pub(crate) struct AppState {
/// Loaded once at startup (`load_hives`); never mutated, so an
/// `Arc` clone per request is the whole synchronization story.
pub(crate) hives: Arc<Vec<HiveEntry>>,
/// Loaded once at startup (`load_links`); same synchronization story
/// as `hives`.
pub(crate) links: Arc<Vec<ServiceLink>>,
/// `None` when this deployment wired up no swarm queue — the only
/// state in which `/api/hives/status` cannot answer at all. A queue
/// that is merely *unreachable* still yields a reader, because
/// `async-nats` reconnects underneath it.
pub(crate) status: Option<Arc<status::StatusReader>>,
/// Publishes the agent set this swarm declares for each hive.
///
/// `None` in exactly the state `status` is: no swarm queue was wired
/// up, so there is nowhere to publish a declaration to. Shares that
/// reader's connection rather than opening a second one.
pub(crate) wanted: Option<Arc<wanted::WantedWriter>>,
/// Per-agent status, sharing `status`'s connection — same "one queue
/// connection, several consumers" rationale as `wanted` above. `None`
/// in exactly the state `status` is.
pub(crate) agent_status: Option<Arc<agent_status::AgentStatusReader>>,
/// Per-agent icons, sharing `status`'s connection — same "one queue
/// connection, several consumers" rationale as `agent_status` above,
/// and `None` in exactly the state `status` is.
pub(crate) agent_icons: Option<Arc<agent_icon::AgentIconReader>>,
/// Each agent's published `AGENTS.md`, on `status`'s connection like
/// `agent_icons`, and `None` in exactly the state `status` is.
pub(crate) agents_md: Option<Arc<agents_md::AgentsMdReader>>,
/// What each agent last reported running on, from its turn-state
/// headers on `status`'s connection. `None` in exactly the state
/// `status` is.
pub(crate) reported_inference: Option<Arc<inference::Reported>>,
/// Whether each agent last said it needs a login, from its turn-state
/// headers on `status`'s connection. `None` in exactly the state
/// `status` is.
pub(crate) agent_login: Option<Arc<agent_login::Reported>>,
/// The swarm-level job graph, wrapped in its
/// [`hive_jobq::scheduler::Scheduler`] now that something drives it
/// (`spawn_jobq_worker`) — the graph alone was enough for the
/// read-only endpoints, the scheduler is what a `claim_next` loop
/// needs. `std::sync::Mutex`, not `tokio`'s — every lock scope below
/// is synchronous (no `.await` while held). Always present, never
/// gated on the swarm queue: this is process state, not something
/// read over the network.
pub(crate) jobq: Arc<Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>>,
/// The identity store's only reader, for `GET /api/agents`.
///
/// `None` when no bridge is wired up, which is the one state in which
/// the roster cannot be answered at all — the same shape as `status`
/// above, and for the same reason: an empty roster and an unreadable
/// one are different facts.
///
/// ⚠️ The two `/api/agents` verbs treat this differently on purpose.
/// **POST does not need it**: creation only queues a job, and
/// whether a bridge exists is `run_swarm_node`'s concern (it holds its
/// own clone from `spawn_jobq_worker`), so a request still queues
/// cleanly on a bridge-less host and fails loud once claimed. POST reads
/// it only for a name that breaks a naming rule, to tell a new agent
/// from an existing one, and warns when it cannot (`name_verdict`). **GET
/// has nowhere to defer to** — there is no job, only an answer it
/// either has or does not. `POST /api/forge/users/{name}/admin` reads
/// it for every name and refuses when it cannot (`admin_unless_agent`).
pub(crate) auth: Option<Arc<auth::AuthBridge>>,
/// HMAC secret for swarm-wide forge webhooks, loaded once at startup.
/// `None` when it could not be read or created — the webhook endpoint
/// then refuses every delivery with 503 rather than admitting one it
/// cannot verify. Deliberately not fatal to startup: nothing is
/// registered against that endpoint yet, and the rest of this daemon's
/// surface is unaffected. See [`load_or_generate_secret`](crate::webhook::load_or_generate_secret).
///
/// `Arc<str>`, not `Arc<String>`: the value is never mutated after
/// startup, and this way `as_deref()` yields the `&str` the verifier
/// takes without a second hop through `String`.
pub(crate) webhook_secret: Option<Arc<str>>,
/// Last successful `agent-configs/*` scan, kept current by
/// [`config_pr::spawn`]. Empty while this daemon has no forge access —
/// same "absent means don't ask" shape as `status` and `forge`, not a
/// startup failure — and filled when the forge client is.
pub(crate) config_prs: Late<Arc<config_pr::ConfigPrCache>>,
/// The swarm's human display name (`services.hyperhive.swarm.name`),
/// loaded once at startup (`load_swarm_name`). `None` when the
/// operator never set it — a swarm without a display name is a
/// supported, if less friendly, state, not a startup failure. Same
/// `Arc<str>` rationale as `webhook_secret`: never mutated, so a
/// clone per request is just a refcount bump.
pub(crate) swarm_name: Option<Arc<str>>,
/// The forge client itself, for the read-only repo/issue-report
/// routes (`GET /api/repos`, `GET /api/repos/{org}/{repo}/issue-report`)
/// and `POST /api/forge/users/{name}/admin` — distinct from `config_prs`, which holds a *cache* built off this
/// same client rather than the client. Empty while this daemon has no
/// forge access, and filled the moment `forge::enable` builds the client.
pub(crate) forge: Late<Arc<forge::Client>>,
/// Held by `create_agent` from reading where a name is placed until its
/// graph is queued, and by `set_agent_state` from that read until its
/// declaration is written, so two placements of one name cannot both
/// find it unplaced. `tokio`'s mutex, unlike `jobq`'s: the read awaits
/// the queue.
pub(crate) create_gate: Arc<tokio::sync::Mutex<()>>,
/// The store the linked-account routes read and write.
pub(crate) accounts: linked_accounts::Accounts,
/// The review classifier try-outs started here, until their results
/// expire.
pub(crate) try_outs: review_classifier::TryOuts,
/// The requested reviews running or waiting.
pub(crate) reviews: Arc<review_classifier::review::Reviews>,
/// The review classifier's finished runs.
pub(crate) runs: review_classifier::runs::Runs,
/// The swarm builder's finished builds.
pub(crate) builds: builder::runs::Builds,
/// Grafana's alerts API, for `GET /api/alerts`. `None` when no Grafana URL
/// is configured.
pub(crate) alerts: Option<Arc<alerts::Grafana>>,
}
/// Why the status route answers 503 rather than an empty list.
///
/// "I cannot reach the store" and "every hive is silent" are different
/// answers, and rendering the second when the first is true is exactly
/// the smoothing this endpoint exists to avoid — a caller would draw a
/// swarm-wide outage out of a local one. The cause is carried in the
/// body because a bare 503 on an operator-facing diagnostic is how a
/// misconfiguration costs an afternoon; it is a queue/JetStream error
/// string, and this surface is already behind the swarm's SSO.
///
/// It travels in `detail` of an RFC 9457 `application/problem+json` body
/// rather than as a bare string, so the cause is an addressable field
/// instead of the whole payload — see `error_problem` below.
pub(crate) struct StatusUnavailable(pub(crate) String);
impl axum::response::IntoResponse for StatusUnavailable {
fn into_response(self) -> axum::response::Response {
error_problem(axum::http::StatusCode::SERVICE_UNAVAILABLE, &self.0).into_response()
}
}
/// Every error this daemon returns, in one shape.
///
/// RFC 9457 `application/problem+json` is the hive-wide contract for HTTP
/// error bodies (`docs/process/conventions.md`), and the operator UIs read `detail`
/// for display. A bare string forces the reader to treat the entire body as
/// the message, which is the difference between a UI that can offer "copy the
/// cause" and one that can only dump a response.
pub(crate) fn error_problem(
status: axum::http::StatusCode,
detail: &str,
) -> problem_details::ProblemDetails {
problem_details::ProblemDetails::from_status_code(status).with_detail(detail)
}
/// The status a [`wanted::WantedWriter`], [`agent_icon::AgentIconReader`] or
/// `schedules` queue failure should answer with.
///
/// `WantedWriter::view`/`set` call `ensure_connected` internally and
/// propagate through `anyhow`, so the concrete
/// [`swarm_queue_client::Error`] survives underneath but is not the type in
/// hand — `swarm_queue_client`'s own doc says a caller "gets a type it can
/// match on", which means downcasting back to it rather than string-matching
/// the rendered message. `NotConnected` is a retryable condition (the queue
/// is reconnecting, or hasn't yet) and every other queue-backed route here
/// already answers 503 for it; anything else is a genuine failure to publish
/// or read, which stays 500.
pub(crate) fn wanted_error_status(e: &anyhow::Error) -> axum::http::StatusCode {
match e.downcast_ref::<swarm_queue_client::Error>() {
Some(swarm_queue_client::Error::NotConnected(_)) => {
axum::http::StatusCode::SERVICE_UNAVAILABLE
}
_ => axum::http::StatusCode::INTERNAL_SERVER_ERROR,
}
}
#[cfg(test)]
pub(crate) mod tests {
use super::*;
/// The 503 this route returns is what the operator UIs' shared error
/// component renders, so assert the RENDERED response rather than the
/// `problem_details` crate: the contract a UI depends on is the content
/// type plus a `detail` it can address, and a handler that built the
/// value and returned it as a bare string would satisfy any test
/// written against the type alone.
#[tokio::test]
async fn status_unavailable_renders_problem_json_with_the_cause_in_detail() {
use axum::response::IntoResponse as _;
let cause = "listing status bucket keys: timed out";
let resp = StatusUnavailable(cause.to_owned()).into_response();
assert_eq!(resp.status(), axum::http::StatusCode::SERVICE_UNAVAILABLE);
let ct = resp
.headers()
.get(axum::http::header::CONTENT_TYPE)
.and_then(|v| v.to_str().ok())
.unwrap_or_default()
.to_owned();
assert!(
ct.starts_with("application/problem+json"),
"RFC 9457 media type, got {ct:?}"
);
let bytes = axum::body::to_bytes(resp.into_body(), 64 * 1024)
.await
.expect("body reads");
let v: serde_json::Value = serde_json::from_slice(&bytes).expect("problem+json parses");
assert_eq!(v["status"], 503);
// The cause is an addressable field, not the entire payload — that
// distinction is the point of the change, so it is what is asserted.
assert_eq!(v["detail"], cause);
}
/// Drives `SwarmNodeKind::CreateRepo` through the real
/// `hive_jobq::scheduler::Scheduler` claim → run → complete path,
/// rather than only through `create_agent`'s endpoint test (there
/// isn't one — the endpoint itself is thin, insert-and-return; the
/// interesting behavior is in `run_swarm_node`'s executor arm, which
/// this exercises directly).
///
/// Deliberately offline: with the forge URL unset,
/// `forge::Client::from_env` returns `Ok(None)` (see that module's doc
/// comment), so this exercises the whole claim → run → complete path
/// through `hive_jobq::scheduler::Scheduler` without a real forge
/// server — at the cost of only ever observing the
/// graceful-absence-is-failure branch here. The happy path needs an
/// actual forge instance and isn't something a unit test in this crate
/// can reach.
///
/// SAFETY: single-threaded mutation of the `forge` env var this
/// test itself owns, restored before returning — no other test in this
/// crate reads them.
/// One roster, one agent name, two hives — the accepted one and a
/// typo. Built per test so neither can see the other's queue.
/// The scheduler handle these tests hold onto so they can assert on
/// what the endpoint queued. Aliased because the full type is three
/// nested generics deep and reads worse inline than named.
pub(crate) type SharedSched = std::sync::Arc<
std::sync::Mutex<
hive_jobq::scheduler::Scheduler<
crate::jobq::SwarmNodeKind,
crate::jobq::SwarmResourceKind,
>,
>,
>;
pub(crate) fn state_with_roster() -> (super::AppState, SharedSched) {
let sched =
std::sync::Arc::new(std::sync::Mutex::new(hive_jobq::scheduler::Scheduler::new(
hive_jobq::Graph::new(),
hive_jobq::resources::ResourceTable::new(),
)));
let state = super::AppState {
hives: std::sync::Arc::new(vec![HiveEntry {
name: "pr1ma".to_owned(),
domain: "pr1ma.example".to_owned(),
dashboard: "pr1ma.hives.example".to_owned(),
}]),
links: std::sync::Arc::new(Vec::new()),
status: None,
// No queue, for the same reason as `status`: these tests build
// the creation graph and read its shape, and never run the node
// that would write a declaration.
wanted: None,
agent_status: None,
agent_icons: None,
agents_md: None,
reported_inference: None,
agent_login: None,
jobq: std::sync::Arc::clone(&sched),
webhook_secret: None,
config_prs: std::sync::Arc::default(),
swarm_name: None,
// No bridge: these tests drive agent *creation*, which queues a
// job and consults one only for a rule-breaking name. The roster
// read is the verb that needs it, and it has its own test below.
auth: None,
forge: std::sync::Arc::default(),
create_gate: std::sync::Arc::default(),
accounts: crate::linked_accounts::Accounts::Fake(std::sync::Arc::default()),
try_outs: crate::review_classifier::TryOuts::default(),
reviews: std::sync::Arc::default(),
runs: crate::review_classifier::runs::Runs::Live(None),
builds: crate::builder::runs::Builds::Live(None),
alerts: None,
};
(state, sched)
}
/// `wanted_error_status` as a pure function first: build the exact
/// `swarm_queue_client::Error::NotConnected` `ensure_connected` raises,
/// wrap it the same way `?` inside `WantedWriter::view`/`set` does, and
/// confirm it downcasts back to the status those routes are meant to
/// answer.
#[test]
fn not_connected_downcasts_to_503() {
let e: anyhow::Error =
swarm_queue_client::Error::NotConnected(async_nats::connection::State::Pending).into();
assert_eq!(
super::wanted_error_status(&e),
axum::http::StatusCode::SERVICE_UNAVAILABLE
);
}
/// Invert-proof for the test above: an `anyhow::Error` that does not
/// wrap a `swarm_queue_client::Error` at all (the shape a genuine write
/// or decode failure takes — `apply`'s "declaration is not decodable"
/// context, for instance) must NOT downcast to `NotConnected`, and must
/// stay 500. Without this, a `wanted_error_status` that answered 503
/// unconditionally would still pass the test above.
#[test]
fn an_unrelated_error_stays_500() {
let e = anyhow::anyhow!("the hive's current declaration is not decodable");
assert_eq!(
super::wanted_error_status(&e),
axum::http::StatusCode::INTERNAL_SERVER_ERROR
);
}
/// Second invert-proof, closer to the actual bug this closes: a
/// *different* `swarm_queue_client::Error` variant (a real connect
/// failure, not `NotConnected`) also stays 500 — proving the match
/// looks at the variant, not just at "was this crate's error type
/// involved at all".
#[test]
fn a_different_queue_client_error_variant_stays_500() {
let e: anyhow::Error = swarm_queue_client::Error::Connect {
url: "nats://queue.example:4222".to_owned(),
source: async_nats::ConnectErrorKind::TimedOut.into(),
}
.into();
assert_eq!(
super::wanted_error_status(&e),
axum::http::StatusCode::INTERNAL_SERVER_ERROR
);
}
}