swarm: mint, publish and login-verify an agent's store identity at create

`swarm/agents/<agent>/bao-mtls` did not exist, and neither did any
per-agent identity at the secret store: `policy::agent_object_name`,
`render_agent` and `render_agent_with_queue` had been written and never
called outside their own tests. An agent's only "per-agent" secret today
is read under the HIVE's certificate, through a wide grant on
`swarm/agents/*` — so "per-agent" was presentational.

The swarm now mints the certificate, so no hive ever needs the capability
to mint one. `swarm-controller` is the service that does it: it already
logs in to the store, and its existing grant already covers exactly the
three objects written here (`create/update` on
`secret/data/swarm/agents/*`, `sys/policies/acl/hive-*` and
`auth/cert/certs/hive-*`). No new bao grant, and nothing co-located — a
cert-auth role pins its authority by value, per role, so the controller
issues from its own CA on its own host and pins that CA in the role it
writes. No existing role changes.

The mint node does not report success on a write. After publishing it
connects again, with the leaf it just issued and under the role it just
wrote, and reads the path back — so the policy, the role, the common name
and the leaf are exercised in production on every agent creation. A
certificate this code mints that the role this code writes will not accept
turns the job node red at creation time instead of surfacing later as an
agent container that cannot start.

`TriggerDeploy` gains an `after_any` edge on the mint, not `after_ok`: a
hive cannot pass down a certificate the swarm has not published, but a
host with no authority configured must still create agents exactly as it
does today.

The private key is generated in memory and never written to disk on the
controller — `SecretStore::connect_with_identity` takes the PEM the minter
is already holding, so nothing is written out purely to be logged in with.

Refs #4137

swarm-secret-client/src/client.rs

@@ -112,7 +112,37 @@ impl SecretStore {
         let mut pem = read_file(ENV_CLIENT_CERT, &settings.cert_path)?;
         pem.push(b'\n');
         pem.extend_from_slice(&read_file(ENV_CLIENT_KEY, &settings.key_path)?);
-        let identity = reqwest::Identity::from_pem(&pem).map_err(Error::Tls)?;
+        Self::connect_with_identity(settings, &pem, cert_role, cert_mount).await
+    }
+
+    /// Log in at the store `settings` names, presenting an identity the caller
+    /// is already holding rather than one named by the environment.
+    ///
+    /// `identity_pem` is the certificate and its private key concatenated into
+    /// one PEM blob — the shape [`reqwest::Identity::from_pem`] takes, and the
+    /// shape [`SecretStore::connect`] builds out of the two files
+    /// [`ENV_CLIENT_CERT`] and [`ENV_CLIENT_KEY`] name. Only the address and
+    /// the CA are read out of `settings` here; its two identity paths are not
+    /// touched.
+    ///
+    /// This is what a **minter** needs. A process that has just issued a leaf
+    /// holds the bytes, and the safest place for a freshly minted private key
+    /// is the memory it was generated in: writing it to a file purely so that
+    /// a `Settings` could name it would put a key on disk for no other reason
+    /// than to log in with it once.
+    ///
+    /// # Errors
+    /// [`Error::Tls`] when `identity_pem` is not a usable certificate/key
+    /// pair, [`Error::Settings`] when the address will not parse, and
+    /// [`Error::Vault`] when the store refuses the login — which is what a
+    /// certificate no cert-auth role accepts looks like from here.
+    pub async fn connect_with_identity(
+        settings: &Settings,
+        identity_pem: &[u8],
+        cert_role: &str,
+        cert_mount: &str,
+    ) -> Result<Self, Error> {
+        let identity = reqwest::Identity::from_pem(identity_pem).map_err(Error::Tls)?;
 
         let mut builder = VaultClientSettingsBuilder::default();
         builder

swarm-secret-client/src/lib.rs

@@ -5,8 +5,8 @@
 //! the rules every path obeys ([`path`]), the translation from this
 //! deployment's environment into a logged-in client ([`client`]), and, per kind
 //! of secret, the path it lives at together with the fields it holds
-//! ([`matrix`], [`queue`]). Each of those is a thing the controller and a hive
-//! must say identically, so it is said once here.
+//! ([`matrix`], [`queue`], [`mtls`]). Each of those is a thing the controller
+//! and a hive must say identically, so it is said once here.
 //!
 //! [`policy`] is the same kind of agreement seen from the other side: which of
 //! those paths a given principal's own token may read — a hive's, and an
@@ -18,9 +18,13 @@
 //! [`client`] is deliberately ignorant of all of it: it moves whatever type a
 //! caller names, so a second kind of secret is a new module beside [`matrix`]
 //! and not another field on a struct shared with it.
+//!
+//! [`mtls`] is the one module about reaching the store rather than about a
+//! value inside it, and its doc explains why that is not circular.
 
 pub mod client;
 pub mod matrix;
+pub mod mtls;
 pub mod path;
 pub mod policy;
 pub mod queue;

swarm-secret-client/src/mtls.rs

@@ -0,0 +1,173 @@
+//! The mTLS agreement: where an agent's own identity **at the store** lives,
+//! and what the object at that path holds.
+//!
+//! The sibling of [`crate::matrix`] and [`crate::queue`], and the one that is
+//! about reaching the store rather than about something kept inside it. An
+//! agent's client certificate is minted at swarm level
+//! (`swarm-controller::agent_identity`) and published here; the agent's hive
+//! collects it under the **hive's** own certificate and hands it into the
+//! container.
+//!
+//! 🔑 The recursion this shape looks like it has — *a credential fetched from
+//! the store that is what opens the store* — is broken by who reads it. The
+//! reader is the hive, never the agent, and a hive already holds its own leaf.
+//! Nothing has to already be an agent in order to obtain an agent's identity.
+
+use serde::{Deserialize, Serialize};
+
+use crate::{
+    Error,
+    path::{Kind, principal_prefix},
+};
+
+/// The path holding `agent`'s client certificate for the store itself.
+///
+/// One path per agent with no further segment below it, unlike
+/// [`crate::matrix::account_path`]: an agent has exactly one identity, and a
+/// second one under a name would be a second principal wearing that name.
+///
+/// # Errors
+/// [`Error::PathSegment`] when `agent` contains anything but `[A-Za-z0-9_-]`,
+/// which is what keeps one agent's name from addressing another's identity.
+pub fn identity_path(agent: &str) -> Result<String, Error> {
+    let prefix = principal_prefix(Kind::Agent, agent)?;
+    Ok(format!("{prefix}/bao-mtls"))
+}
+
+/// What the path holds: the leaf, its private key, and the authority the leaf
+/// was issued from.
+///
+/// All three, because a reader has to reconstruct a usable identity from the
+/// store alone — the same requirement [`crate::queue::Credential`] states for
+/// carrying its client id. The authority rides along so the cert-auth role
+/// and the certificate cannot be delivered from two different sources and
+/// silently disagree; it is public material, unlike the other two fields.
+///
+/// ⚠️ **No `Debug` derive.** See the hand-written impl below: one of these
+/// fields is a private key, and a derived `Debug` would put it in any log line
+/// that ever formatted a node's payload.
+#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
+pub struct Credential {
+    /// The agent's client certificate, PEM. The nix-side reader in an agent's
+    /// hive spells `bao kv get -field=cert`, so this name is load-bearing for
+    /// a reader this crate does not control.
+    pub cert: String,
+
+    /// The private key for [`Credential::cert`], PEM. `bao kv get -field=key`
+    /// on the reading side, and `0600` the moment it lands on disk there.
+    pub key: String,
+
+    /// The authority [`Credential::cert`] was issued from, PEM. Public
+    /// material: it is also what the store keeps inside the agent's cert-auth
+    /// role, by value (see [`crate::client::SecretStore::write_cert_role`]).
+    pub ca: String,
+}
+
+impl std::fmt::Debug for Credential {
+    /// Redacts the key and summarises the two public fields by length.
+    ///
+    /// Hand-written rather than derived because the derive is the failure:
+    /// this type is carried through a job-graph node and an `anyhow` context
+    /// chain, both of which format whatever they are given.
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("Credential")
+            .field("cert", &format_args!("{} bytes of PEM", self.cert.len()))
+            .field("key", &"<redacted>")
+            .field("ca", &format_args!("{} bytes of PEM", self.ca.len()))
+            .finish()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn an_agent_name_lands_under_its_own_principal_prefix() {
+        assert_eq!(
+            identity_path("atlas").expect("a plain name is legal"),
+            "swarm/agents/atlas/bao-mtls"
+        );
+    }
+
+    #[test]
+    fn a_traversal_in_the_agent_name_is_refused() {
+        let e = identity_path("../argus").expect_err("a traversal is not");
+        assert!(matches!(e, Error::PathSegment { kind: "agent", .. }), "{e}");
+    }
+
+    #[test]
+    fn two_agents_never_share_a_path() {
+        assert_ne!(
+            identity_path("atlas").expect("legal"),
+            identity_path("argus").expect("legal")
+        );
+    }
+
+    /// The path sits under the same prefix an agent's policy grants
+    /// (`policy::render_agent`'s stanza is `swarm/agents/<agent>/*`), which is
+    /// what makes an agent able to read its own identity back.
+    #[test]
+    fn an_agents_identity_is_inside_its_own_policy_stanza() {
+        let path = identity_path("atlas").expect("legal");
+        let document = crate::policy::render_agent("atlas").expect("legal");
+        let prefix = format!("{}/data/swarm/agents/atlas/", crate::path::MOUNT);
+        assert!(
+            path.starts_with("swarm/agents/atlas/"),
+            "the identity must sit under the agent's own prefix, got {path}"
+        );
+        assert!(
+            document.contains(&format!("{prefix}*")),
+            "the agent's document must cover {prefix}*, got:\n{document}"
+        );
+    }
+
+    #[test]
+    fn the_object_round_trips_through_the_store_representation() {
+        let c = Credential {
+            cert: "-----BEGIN CERTIFICATE-----\n".to_owned(),
+            key: "-----BEGIN PRIVATE KEY-----\n".to_owned(),
+            ca: "-----BEGIN CERTIFICATE-----\n".to_owned(),
+        };
+        let json = serde_json::to_string(&c).expect("serialises");
+        assert_eq!(
+            serde_json::from_str::<Credential>(&json).expect("deserialises"),
+            c
+        );
+    }
+
+    #[test]
+    fn the_field_names_the_nix_reader_asks_for_are_the_ones_written() {
+        let json = serde_json::to_value(Credential {
+            cert: "leaf".to_owned(),
+            key: "private".to_owned(),
+            ca: "authority".to_owned(),
+        })
+        .expect("serialises");
+        assert_eq!(json["cert"], "leaf");
+        assert_eq!(json["key"], "private");
+        assert_eq!(json["ca"], "authority");
+    }
+
+    /// The property the hand-written `Debug` exists for: a key that reaches a
+    /// log line is a key on a disk somebody else owns.
+    #[test]
+    fn formatting_the_credential_does_not_reveal_the_key() {
+        let rendered = format!(
+            "{:?}",
+            Credential {
+                cert: "leaf".to_owned(),
+                key: "SUPER-SECRET-KEY-MATERIAL".to_owned(),
+                ca: "authority".to_owned(),
+            }
+        );
+        assert!(
+            !rendered.contains("SUPER-SECRET-KEY-MATERIAL"),
+            "the key must not survive formatting, got {rendered}"
+        );
+        assert!(
+            rendered.contains("<redacted>"),
+            "and the reader must be told it was withheld, got {rendered}"
+        );
+    }
+}