feat(#3124): publish the agent set the swarm declares for each hive
The hive-side loop landed without anything to converge to: nothing wrote `$KV.hive-wanted.<hive>`, so in production only the "no key" branch ran. This is the writer. `WantedWriter` mirrors `StatusReader` — that module reads what hives report, this one writes what they are told, so it holds a client rather than a bucket handle and resolves the store on first use. It shares the status reader's connection: the controller has exactly one by design, and a second connect would double the auth-callout traffic and give the two paths independent reconnect state. The value under a hive's key is the map of every agent on that hive, so a plain `put` of a single-agent change would drop a concurrent change to a different agent, with only one revision of history to not recover from. Writes are read-modify-write against the entry revision, and only `WrongLastRevision` / `AlreadyExists` count as a lost race — every other error returns immediately rather than spinning the retry loop and then blaming a concurrent writer that never existed. `apply` is split out and tested because it holds the invariant: declaring one agent preserves the rest, and a current value that will not decode is an error rather than a fresh start. Overwriting a document nobody can read discards every other agent's declaration. Two routes, no swarmctl verb and no jobq node: `create_agent` needs a graph because it is multi-step, and one CAS'd write is not. `build_app` is extracted from `main` in the same change because `main` sat at exactly the `too_many_lines` limit, so adding an endpoint tripped a lint about the startup sequence. The route list is the part that grows.
This commit is contained in:
atlas
parent
066f31b58b
commit
76d5871d20
3 changed files with 424 additions and 9 deletions
|
|
@ -47,6 +47,7 @@ mod issue_report;
|
|||
mod otel_http_client;
|
||||
mod status;
|
||||
mod vcs_metrics;
|
||||
mod wanted;
|
||||
mod webhook;
|
||||
|
||||
/// Node payload for the swarm-level job graph. Named `Swarm*` rather than
|
||||
|
|
@ -429,6 +430,12 @@ struct AppState {
|
|||
/// that is merely *unreachable* still yields a reader, because
|
||||
/// `async-nats` reconnects underneath it.
|
||||
status: Option<Arc<status::StatusReader>>,
|
||||
/// Publishes the agent set this swarm declares for each hive.
|
||||
///
|
||||
/// `None` in exactly the state `status` is: no swarm queue was wired
|
||||
/// up, so there is nowhere to publish a declaration to. Shares that
|
||||
/// reader's connection rather than opening a second one.
|
||||
wanted: Option<Arc<wanted::WantedWriter>>,
|
||||
/// The swarm-level job graph, wrapped in its
|
||||
/// [`hive_jobq::scheduler::Scheduler`] now that something drives it
|
||||
/// (`spawn_jobq_worker`) — the graph alone was enough for the
|
||||
|
|
@ -681,6 +688,150 @@ fn error_problem(status: axum::http::StatusCode, detail: &str) -> problem_detail
|
|||
problem_details::ProblemDetails::from_status_code(status).with_detail(detail)
|
||||
}
|
||||
|
||||
/// The state to declare for one agent.
|
||||
#[derive(Debug, Deserialize, ToSchema)]
|
||||
struct SetAgentStateRequest {
|
||||
/// Typed as a string in the schema only — the parse is the real enum, so
|
||||
/// a value this build does not know is a 400 rather than a field that
|
||||
/// silently does nothing.
|
||||
#[schema(value_type = String, example = "up")]
|
||||
state: swarm_queue_client::wanted::AgentState,
|
||||
}
|
||||
|
||||
/// One agent's line in a hive's declaration.
|
||||
#[derive(Clone, Debug, Serialize, ToSchema)]
|
||||
struct AgentDeclaration {
|
||||
agent: String,
|
||||
state: String,
|
||||
}
|
||||
|
||||
fn render(declaration: &swarm_queue_client::wanted::HiveWanted) -> Vec<AgentDeclaration> {
|
||||
declaration
|
||||
.agents
|
||||
.iter()
|
||||
.map(|(agent, wanted)| AgentDeclaration {
|
||||
agent: agent.clone(),
|
||||
state: wanted.state.as_str().to_owned(),
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The declaration writer, sharing the status reader's connection.
|
||||
///
|
||||
/// The controller holds exactly one queue connection by design — see
|
||||
/// `StatusReader::queue_client`. A second connect would double the
|
||||
/// auth-callout traffic and give the two paths independent reconnect state,
|
||||
/// so one could be serving while the other was still down.
|
||||
fn wanted_writer(status: Option<&Arc<status::StatusReader>>) -> Option<Arc<wanted::WantedWriter>> {
|
||||
status.map(|s| Arc::new(wanted::WantedWriter::new(s.queue_client())))
|
||||
}
|
||||
|
||||
/// Both handlers below take the same hive name and reject it the same way.
|
||||
///
|
||||
/// Reports the status and the detail rather than a rendered
|
||||
/// `ProblemDetails`: that type is 232 bytes, which makes every `Result` in
|
||||
/// this path pay for the error case it usually does not take. The handlers
|
||||
/// render at the boundary, where the body is actually needed.
|
||||
fn declaration_target(
|
||||
state: &AppState,
|
||||
hive: &str,
|
||||
) -> Result<(Arc<wanted::WantedWriter>, String), (axum::http::StatusCode, String)> {
|
||||
let writer = state.wanted.clone().ok_or_else(|| {
|
||||
(
|
||||
axum::http::StatusCode::SERVICE_UNAVAILABLE,
|
||||
"this deployment wired up no swarm queue, so there is nowhere to publish a declaration"
|
||||
.to_owned(),
|
||||
)
|
||||
})?;
|
||||
let hive = hive_types::Ident::parse(hive)
|
||||
.map_err(|reason| (axum::http::StatusCode::BAD_REQUEST, reason.to_owned()))?
|
||||
.into_string();
|
||||
// Shaped like a hive name, and actually one. Same two checks
|
||||
// `create_agent` makes, for the same reason: a typo otherwise publishes a
|
||||
// declaration under a key no hive will ever read.
|
||||
if !state.hives.iter().any(|h| h.name == hive) {
|
||||
return Err((
|
||||
axum::http::StatusCode::BAD_REQUEST,
|
||||
format!("{hive:?} is not a hive in this swarm"),
|
||||
));
|
||||
}
|
||||
Ok((writer, hive))
|
||||
}
|
||||
|
||||
/// Declare what this swarm wants of one agent on one hive.
|
||||
///
|
||||
/// The whole declaration is returned as published, because the value is the
|
||||
/// hive's entire agent map and a caller that changed one agent still wants to
|
||||
/// render the rest.
|
||||
#[utoipa::path(
|
||||
put,
|
||||
path = "/api/hives/{hive}/agents/{agent}/state",
|
||||
params(
|
||||
("hive" = String, Path, description = "hive whose declaration this is"),
|
||||
("agent" = String, Path, description = "agent to declare"),
|
||||
),
|
||||
request_body = SetAgentStateRequest,
|
||||
responses(
|
||||
(status = 200, description = "the declaration as now published", body = Vec<AgentDeclaration>),
|
||||
(status = 400, description = "a name is not an identifier, the hive is not in this swarm, or the state is unknown (problem+json)", body = String),
|
||||
(status = 503, description = "no swarm queue is wired up (problem+json)", body = String),
|
||||
(status = 500, description = "the declaration could not be published (problem+json)", body = String),
|
||||
),
|
||||
tag = "agents"
|
||||
)]
|
||||
async fn set_agent_state(
|
||||
State(state): State<AppState>,
|
||||
axum::extract::Path((hive, agent)): axum::extract::Path<(String, String)>,
|
||||
Json(req): Json<SetAgentStateRequest>,
|
||||
) -> Result<Json<Vec<AgentDeclaration>>, problem_details::ProblemDetails> {
|
||||
let (writer, hive) =
|
||||
declaration_target(&state, &hive).map_err(|(s, d)| error_problem(s, &d))?;
|
||||
let agent = hive_types::Ident::parse(&agent)
|
||||
.map_err(|reason| error_problem(axum::http::StatusCode::BAD_REQUEST, reason))?
|
||||
.into_string();
|
||||
|
||||
let declaration = writer.set(&hive, &agent, req.state).await.map_err(|e| {
|
||||
tracing::warn!(hive = %hive, agent = %agent, error = %format!("{e:#}"), "declaring agent state failed");
|
||||
error_problem(
|
||||
axum::http::StatusCode::INTERNAL_SERVER_ERROR,
|
||||
&format!("{e:#}"),
|
||||
)
|
||||
})?;
|
||||
Ok(Json(render(&declaration)))
|
||||
}
|
||||
|
||||
/// What this swarm currently declares for a hive.
|
||||
///
|
||||
/// Read back from the bucket rather than from a second copy kept here: the
|
||||
/// published value is the record.
|
||||
#[utoipa::path(
|
||||
get,
|
||||
path = "/api/hives/{hive}/wanted",
|
||||
params(("hive" = String, Path, description = "hive whose declaration to read")),
|
||||
responses(
|
||||
(status = 200, description = "the declaration, empty when nothing is published yet", body = Vec<AgentDeclaration>),
|
||||
(status = 400, description = "not an identifier, or not a hive in this swarm (problem+json)", body = String),
|
||||
(status = 503, description = "no swarm queue is wired up (problem+json)", body = String),
|
||||
(status = 500, description = "the declaration could not be read (problem+json)", body = String),
|
||||
),
|
||||
tag = "agents"
|
||||
)]
|
||||
async fn get_hive_wanted(
|
||||
State(state): State<AppState>,
|
||||
axum::extract::Path(hive): axum::extract::Path<String>,
|
||||
) -> Result<Json<Vec<AgentDeclaration>>, problem_details::ProblemDetails> {
|
||||
let (writer, hive) =
|
||||
declaration_target(&state, &hive).map_err(|(s, d)| error_problem(s, &d))?;
|
||||
let declaration = writer.view(&hive).await.map_err(|e| {
|
||||
tracing::warn!(hive = %hive, error = %format!("{e:#}"), "reading the declaration failed");
|
||||
error_problem(
|
||||
axum::http::StatusCode::INTERNAL_SERVER_ERROR,
|
||||
&format!("{e:#}"),
|
||||
)
|
||||
})?;
|
||||
Ok(Json(declaration.as_ref().map(render).unwrap_or_default()))
|
||||
}
|
||||
|
||||
/// What each hive last said about itself, read from the swarm queue at
|
||||
/// request time.
|
||||
///
|
||||
|
|
@ -1328,6 +1479,7 @@ async fn main() -> Result<()> {
|
|||
let state = AppState {
|
||||
hives: Arc::new(load_hives()),
|
||||
links: Arc::new(load_links()),
|
||||
wanted: wanted_writer(status.as_ref()),
|
||||
status,
|
||||
jobq,
|
||||
webhook_secret,
|
||||
|
|
@ -1337,6 +1489,19 @@ async fn main() -> Result<()> {
|
|||
forge: state_forge,
|
||||
};
|
||||
|
||||
let app = build_app(state);
|
||||
axum::serve(listener, app)
|
||||
.await
|
||||
.context("serving swarm-controller")
|
||||
}
|
||||
|
||||
/// Every route this daemon serves, wired to its state.
|
||||
///
|
||||
/// Split out of `main` because the route list is the part that grows, and
|
||||
/// `main` sat exactly on the `too_many_lines` limit — adding an endpoint
|
||||
/// tripped a lint about the startup sequence, which is not where the change
|
||||
/// was. A new route now costs one line here and none there.
|
||||
fn build_app(state: AppState) -> axum::Router {
|
||||
let (router, api) = OpenApiRouter::<AppState>::with_openapi(ApiDoc::openapi())
|
||||
.routes(routes!(health))
|
||||
.routes(routes!(get_hives))
|
||||
|
|
@ -1349,6 +1514,8 @@ async fn main() -> Result<()> {
|
|||
.routes(routes!(get_config_prs))
|
||||
.routes(routes!(create_agent))
|
||||
.routes(routes!(get_agents))
|
||||
.routes(routes!(set_agent_state))
|
||||
.routes(routes!(get_hive_wanted))
|
||||
.routes(routes!(issue_report::get_repos))
|
||||
.routes(routes!(issue_report::get_issue_report_all))
|
||||
.routes(routes!(issue_report::get_issue_report))
|
||||
|
|
@ -1358,15 +1525,12 @@ async fn main() -> Result<()> {
|
|||
// the nix store (see the module doc comment above). `api` is
|
||||
// `Clone`; each request gets its own owned copy for `Json` to
|
||||
// serialize, same as `hive-c0re::dashboard::serve`.
|
||||
let app = router
|
||||
router
|
||||
.route(
|
||||
"/api/openapi.json",
|
||||
get(move || async move { Json(api.clone()) }),
|
||||
)
|
||||
.with_state(state);
|
||||
axum::serve(listener, app)
|
||||
.await
|
||||
.context("serving swarm-controller")
|
||||
.with_state(state)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
|
@ -1455,6 +1619,9 @@ mod tests {
|
|||
}]),
|
||||
links: std::sync::Arc::new(Vec::new()),
|
||||
status: None,
|
||||
// No queue, for the same reason as `status`: these tests drive
|
||||
// agent creation, which publishes no declaration.
|
||||
wanted: None,
|
||||
jobq: std::sync::Arc::clone(&sched),
|
||||
webhook_secret: None,
|
||||
config_prs: None,
|
||||
|
|
|
|||
Loading…
Reference in a new issue