swarmctl: re-mint an existing agent's store identity

Agent creation at swarm level is event-driven and nothing sweeps for
agents missing a credential, so an agent created before a credential
joined the mint never receives one -- nothing comes back around to it.
Without a way to re-run the mint by hand, the only route to giving an
existing agent its queue credential would be to delete and recreate the
agent.

POST /api/agents/{name}/identity enqueues the same MintAgentIdentity
node POST /api/agents declares, rather than writing inline: a second
code path that mints an identity is a second place for the four strings
that have to agree to disagree. swarmctl agent mint-identity is the
operator end, the same POST-and-print-the-node-id shape agent create
already has.

--hive is required on both ends. Neither the CLI nor the controller
keeps a roster of which agent runs where, and the credentials this mints
name a hive, so a default would be a guess that hands an agent subjects
on a hive it does not run on.

Documents the backfill as a runbook step, and fills in the renewal cell
the credential matrix requires for the new row.

swarm-controller/src/agent_identity.rs

@@ -8,14 +8,12 @@
 //! store, and its grant already covers exactly the objects written here
 //! (`swarm-bao.nix`'s `controllerPolicyText`: `create/update` on
 //! `secret/data/swarm/agents/*`, on `sys/policies/acl/hive-*`, and on
-//! `auth/cert/certs/hive-*`). No new authority is asked for anywhere — the
-//! agent's queue secret lives under the same `swarm/agents/<agent>` prefix the
-//! certificate does, which is why adding it costs no grant on either side.
+//! `auth/cert/certs/hive-*`). No new authority is asked for anywhere.
 //!
-//! **Two credentials, deliberately unrelated.** The certificate is how the
-//! agent reaches the store; the queue secret is how it identifies itself to the
-//! swarm queue. The second is not derived from the first, so renewing either is
-//! a question that can be answered without reference to the other.
+//! **Two credentials, deliberately unrelated.** The certificate reaches the
+//! store; the queue secret identifies the agent to the swarm queue. Both sit
+//! under `swarm/agents/<agent>`, so neither costs a grant — but the second is
+//! not derived from the first, so either renews without reference to the other.
 //!
 //! Four separate strings have to agree before an agent can authenticate: the
 //! policy's name, the cert-auth role's name, the certificate's common name,
@@ -247,14 +245,12 @@ fn generate_queue_secret() -> Result<String> {
 /// names the policy, so the other order leaves a window in which it points at
 /// nothing.
 ///
-/// ⚠️ **Step 3 is idempotent and step 2 is not.** Re-running this function
-/// re-mints the agent's certificate — a fresh leaf the agent picks up on its
-/// next boot — but leaves an existing queue secret exactly as it is. The
-/// asymmetry is deliberate: an agent holds its queue secret in a live
-/// connection, so replacing it would drop that agent off the queue until it
-/// reconnected, and this function is re-run deliberately (by the backfill
-/// route) against agents that are already running. Nothing here rotates a
-/// queue secret; revoking one means deleting the path.
+/// ⚠️ **Step 3 is idempotent and step 2 is not.** Re-running re-mints the
+/// certificate — a fresh leaf the agent picks up on its next boot — but leaves
+/// an existing queue secret alone. An agent holds that secret in a live
+/// connection, and this function is re-run deliberately against agents that
+/// are already running, so replacing it would drop them off the queue.
+/// Nothing here rotates one; revoking means deleting the path.
 ///
 /// # Errors
 /// Anything that stops one of those five steps, with the step named. A

swarm-controller/src/main.rs

@@ -1500,6 +1500,111 @@ async fn get_agent_config_pr(
     Ok(Json(cache.get(&name)))
 }
 
+/// Body of `POST /api/agents/{name}/identity`.
+#[derive(Deserialize, ToSchema)]
+struct MintAgentIdentityRequest {
+    /// The hive this agent belongs to.
+    ///
+    /// Required, for the same reason [`CreateAgentRequest`]'s is: the
+    /// credentials this mints name a hive, and the controller has nowhere to
+    /// look one up — agents are created on hives at runtime and this daemon
+    /// keeps no roster of which agent is where. An operator naming the wrong
+    /// one would hand the agent subjects on a hive it does not run on, so it
+    /// is asked for rather than guessed at.
+    hive: String,
+}
+
+/// Success body of `POST /api/agents/{name}/identity`.
+#[derive(Serialize, ToSchema)]
+struct MintAgentIdentityResponse {
+    /// The queued node, so a caller can follow it in the job view.
+    node_id: u64,
+}
+
+/// Re-run the store-identity mint for an agent that already exists.
+///
+/// **This route exists for backfill.** Agent creation is event-driven and
+/// there is no reconcile sweep at swarm level, so an agent created before a
+/// credential was part of the mint never gets one — nothing would ever come
+/// back around to it. Without a way to re-run the node by hand, the only way
+/// to give an existing agent its queue credential would be to delete and
+/// recreate the agent.
+///
+/// A job node rather than an inline write, and specifically the *same* node
+/// `POST /api/agents` declares: a second code path that mints an identity is a
+/// second place for the four strings in [`agent_identity`] to disagree.
+///
+/// ⚠️ **Not idempotent in one respect**: the node re-mints the agent's mTLS
+/// leaf, which the agent picks up on its next boot. Its queue secret is left
+/// alone — see [`agent_identity::mint_and_verify`] for which half is which and
+/// why.
+#[utoipa::path(
+    post,
+    path = "/api/agents/{name}/identity",
+    params(("name" = String, Path, description = "agent name")),
+    request_body = MintAgentIdentityRequest,
+    responses(
+        (status = 200, description = "mint queued", body = MintAgentIdentityResponse),
+        (status = 400, description = "`name` or `hive` is not a valid identifier, or `hive` is not in this swarm (problem+json)", body = String),
+        (status = 500, description = "the job could not be queued (problem+json)", body = String),
+    ),
+    tag = "agents"
+)]
+async fn mint_agent_identity(
+    State(state): State<AppState>,
+    Path(name): Path<String>,
+    Json(req): Json<MintAgentIdentityRequest>,
+) -> Result<Json<MintAgentIdentityResponse>, problem_details::ProblemDetails> {
+    // Both names are interpolated into store paths and policy documents
+    // downstream, so both are validated here as well as there.
+    let agent = hive_types::Ident::parse(&name)
+        .map_err(|reason| error_problem(axum::http::StatusCode::BAD_REQUEST, reason))?
+        .into_string();
+    let hive = hive_types::Ident::parse(&req.hive)
+        .map_err(|reason| error_problem(axum::http::StatusCode::BAD_REQUEST, reason))?
+        .into_string();
+    if !state.hives.iter().any(|h| h.name == hive) {
+        let known: Vec<&str> = state.hives.iter().map(|h| h.name.as_str()).collect();
+        let known = if known.is_empty() {
+            "(none configured)".to_owned()
+        } else {
+            known.join(", ")
+        };
+        return Err(error_problem(
+            axum::http::StatusCode::BAD_REQUEST,
+            &format!("hive {hive:?} is not in this swarm — known hives: {known}"),
+        ));
+    }
+
+    // No reserved-name or collision warnings here, unlike `create_agent`:
+    // those answer "is this name available", and this route is only ever
+    // pointed at a name that was already taken.
+    let mut sched = state
+        .jobq
+        .lock()
+        .unwrap_or_else(std::sync::PoisonError::into_inner);
+    let ids = sched
+        .insert_job(None, |b| {
+            vec![
+                b.node(SwarmNodeKind::MintAgentIdentity {
+                    hive: hive.clone(),
+                    agent: agent.clone(),
+                })
+                .guid(),
+            ]
+        })
+        .map_err(|e| {
+            error_problem(
+                axum::http::StatusCode::INTERNAL_SERVER_ERROR,
+                &e.to_string(),
+            )
+        })?;
+    let [id] = ids[..] else {
+        unreachable!("exactly one handle was asked for");
+    };
+    Ok(Json(MintAgentIdentityResponse { node_id: id.get() }))
+}
+
 /// Every agent with an open config PR, in one response — the bulk
 /// counterpart to [`get_agent_config_pr`]. swarm-ui's config-PR table needs
 /// every agent's status to render, and fetching them one at a time doesn't
@@ -1837,6 +1942,7 @@ fn build_app(state: AppState) -> axum::Router {
         .routes(routes!(get_agent_config_pr))
         .routes(routes!(get_config_prs))
         .routes(routes!(create_agent))
+        .routes(routes!(mint_agent_identity))
         .routes(routes!(get_agents))
         .routes(routes!(get_agents_status))
         .routes(routes!(set_agent_state))