vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-bao: agent certificates issued by a store-generated agent CA

Browse source 
An agent's store identity was signed in swarm-controller's memory by a CA
a controller-host unit generated on disk, and the listener never trusted
that CA. Agent leaves now come from the store itself: a `pki-agents` PKI
mount whose root openbao generates internally, so the agent CA's key
never exists outside the store.

- swarm-bao-agent-pki (new, store host, as the bao granter): enables and
  tunes the mount, generates the root once (guarded on an empty issuer
  list, no replace branch), upserts the `swarm-agent` role (client
  certificates named `hive-agent-*` only, 90 days), caches the CA at
  /var/lib/swarm-bao-tls/agent-ca.pem and composes the listener bundle.
- The listener's tls_client_ca_file is a new listener-client-ca.pem
  (client-ca.pem, then the agent CA). Host cert-auth roles still pin
  client-ca.pem, so an agent leaf satisfies no host role. swarm-bao-certs
  composes the same bundle before openbao starts.
- openbao reads tls_client_ca_file only at start, so when the bundle
  changed after openbao started, swarm-bao-agent-pki restarts
  openbao.service in the container; under `seal = "shamir"` it prints
  the step instead. Once swarm-bao-certs has a cached CA, later boots
  start openbao with it and do not restart.
- The controller policy gains exactly `update` on
  pki-agents/issue/swarm-agent. mint_and_verify now asks that role for
  the leaf (the store generates the key), writes the agent's cert-auth
  role pinning the issuing CA bao returned, and writes the agent's
  policy as render_agent alone: the hive-shared queue credential stanza
  is gone.
- deploy.bao.agentPkiRoleName (must start `swarm-`, asserted with the
  other pki role names); swarm-controller gets
  SWARM_CONTROLLER_AGENT_PKI_MOUNT/_ROLE from the deploy.bao options.

Deleted: swarm-controller-agent-ca and its options (agentCaFile,
agentCaKeyFile), env, LoadCredential entries and assertion;
agent_identity's Authority, rcgen signing and validity window; the
rcgen and time dependencies of swarm-controller (rcgen leaves the
workspace); policy::render_agent_with_queue and its tests. The CN-prefix
assertion policy.rs said was owed is not: agent and host roles pin
different CAs.

Migration is re-creating each agent after deploy; that overwrites the
stale role and policy.

Closes #4756
This commit is contained in:
atlas 2026-09-27 19:30:38 +02:00
commit 6170e74a31
16 changed files with 894 additions and 925 deletions
Download patch file Download diff file Expand all files Collapse all files

453
swarm-controller/src/agent_identity.rs
View file

@ -1,14 +1,14 @@
//! One agent's own identity at the swarm's secret store: minted here,
//! One agent's own identity at the swarm's secret store: issued by the store,
//! published here, granted here — and, before the job node reports success,
//! **used** here.
//!
//! The swarm mints the agent's certificate so that no hive ever needs the
//! capability to mint one; the hive only carries it down. The controller is
//! the swarm-level service that does it because it already logs in to the
//! store, and its grant already covers exactly the objects written here
//! (`swarm-bao.nix`'s `controllerPolicyText`: `create/update` on
//! `secret/data/swarm/agents/*`, on `sys/policies/acl/hive-*`, and on
//! `auth/cert/certs/hive-*`). No new authority is asked for anywhere.
//! The swarm obtains the agent's certificate so that no hive ever needs the
//! capability to obtain one; the hive only carries it down. The controller is
//! the swarm-level service that asks because it already logs in to the store,
//! and its grant covers exactly the calls made here (`swarm-bao.nix`'s
//! `controllerPolicyText`: `update` on the agent PKI role's `issue` path, and
//! `create/update` on `secret/data/swarm/agents/*`, on
//! `sys/policies/acl/hive-*`, and on `auth/cert/certs/hive-*`).
//!
//! **Two credentials, deliberately unrelated.** The certificate reaches the
//! store; the queue secret identifies the agent to the swarm queue. Both sit
@ -22,14 +22,11 @@
//! four — so [`mint_and_verify`] does not finish on a write. See
//! [`read_back_as_agent`].
//!
//! ⚠️ The authority is **not** `/var/lib/swarm-bao-pki/ca-key.pem`. A
//! cert-auth role pins its authority by value, per role (see
//! [`SecretStore::write_cert_role`][swarm_secret_client::SecretStore::write_cert_role]),
//! so a role this daemon writes carries whatever authority this daemon hands
//! it — which is what lets the controller mint from its own CA on its own
//! host, with nothing co-located and no existing role changed.
use std::time::{Duration, SystemTime, UNIX_EPOCH};
//! The authority is the store's own agent CA, generated inside its agent PKI
//! mount; its key never leaves the store. It signs no host leaf, and no host
//! role pins it, so an agent's certificate satisfies only that agent's role.
//! Nothing re-issues a leaf before it expires (the role's `ttl`); until a
//! renewal path exists, an operator re-runs agent creation.
use anyhow::{Context, Result, bail};
use swarm_secret_client::{
@ -37,163 +34,47 @@ use swarm_secret_client::{
client::{DEFAULT_CERT_MOUNT, Settings},
mtls, policy, queue,
};
use time::OffsetDateTime;
/// File holding the authority agent leaves are issued from, as
/// `swarm-controller.nix` names it. Public material.
pub const ENV_AGENT_CA: &str = "SWARM_CONTROLLER_AGENT_CA_FILE";
/// The PKI mount agent leaves are issued from, as `swarm-controller.nix` sets
/// it from `deploy.bao.agentPkiMountPath`.
pub const ENV_AGENT_PKI_MOUNT: &str = "SWARM_CONTROLLER_AGENT_PKI_MOUNT";
/// File holding the private key for [`ENV_AGENT_CA`]. 🩸 A path, never a
/// value — the key's bytes must not reach a unit file or the nix store.
pub const ENV_AGENT_CA_KEY: &str = "SWARM_CONTROLLER_AGENT_CA_KEY_FILE";
/// The role on [`ENV_AGENT_PKI_MOUNT`] agent leaves are issued through, as
/// `swarm-controller.nix` sets it from `deploy.bao.agentPkiRoleName`.
pub const ENV_AGENT_PKI_ROLE: &str = "SWARM_CONTROLLER_AGENT_PKI_ROLE";
/// How long a minted leaf is good for.
///
/// Short enough that a leaked key is not permanent, long enough that the
/// absence of a renewal path is not immediately fatal. Nothing re-mints a leaf
/// today, so until a renewal path lands this is the interval after which an
/// operator re-runs agent creation. It is deliberately far shorter than the ten
/// years
/// `glue-bao-tls.nix` gives the store's own CA: that one is an authority
/// whose reissue invalidates every leaf under it, this one is a leaf.
/// Spelled in hours because `Duration::from_days` is not yet a stable `const
/// fn`; `read_policy`'s `RETRY_WINDOW` is the same workaround.
const LEAF_LIFETIME: Duration = Duration::from_hours(90 * 24);
/// How far a leaf is backdated.
///
/// The verifier is the store, on another machine: a certificate whose
/// `notBefore` is this exact instant is refused outright by a clock a second
/// behind ours, and the resulting error names a validity window rather than a
/// clock.
const CLOCK_SKEW: Duration = Duration::from_mins(5);
/// The authority this daemon issues agent leaves from, loaded once at startup.
///
/// ⚠️ **No `Debug` derive**, and the key field is private: this struct is
/// reachable from `WorkerDeps`, which is formatted nowhere today and is one
/// `#[derive(Debug)]` away from being formatted everywhere.
pub struct Authority {
/// The authority's certificate, PEM. Public material — it is also what
/// goes into each agent's cert-auth role and into each agent's published
/// credential.
ca_pem: String,
/// The authority's private key, PEM. Never logged, never published,
/// never leaves this struct.
key_pem: String,
}
impl Authority {
/// Load the authority from the files [`ENV_AGENT_CA`] and
/// [`ENV_AGENT_CA_KEY`] name, or `None` when this host was given neither.
///
/// `None` is a supported deployment, not a failure: it is the state every
/// controller is in before an operator has turned agent identities on, and
/// `run_swarm_node` reports it as that one node's named failure rather
/// than refusing to start the daemon.
///
/// # Errors
/// When exactly one of the two variables is set — half an authority signs
/// nothing, and silently doing nothing about it is how a host ends up
/// looking configured — or when a named file cannot be read.
pub fn from_env() -> Result<Option<Self>> {
match (
std::env::var_os(ENV_AGENT_CA),
std::env::var_os(ENV_AGENT_CA_KEY),
) {
(None, None) => Ok(None),
(Some(_), None) => bail!(
"{ENV_AGENT_CA} is set but {ENV_AGENT_CA_KEY} is not — an authority with no key signs nothing"
),
(None, Some(_)) => bail!(
"{ENV_AGENT_CA_KEY} is set but {ENV_AGENT_CA} is not — a key with no certificate is not an authority"
),
(Some(ca), Some(key)) => {
let ca_path = ca.to_string_lossy().into_owned();
let key_path = key.to_string_lossy().into_owned();
Ok(Some(Self {
ca_pem: std::fs::read_to_string(&ca_path).with_context(|| {
format!("reading the agent authority {ca_path} (from {ENV_AGENT_CA})")
})?,
key_pem: std::fs::read_to_string(&key_path).with_context(|| {
format!(
"reading the agent authority's key {key_path} (from {ENV_AGENT_CA_KEY})"
)
})?,
}))
}
}
}
/// Build one from PEM already in hand — the constructor a test uses, and
/// the one that keeps [`Authority::from_env`] the only place this process
/// reads a key off disk.
#[cfg(test)]
fn from_pem(ca_pem: String, key_pem: String) -> Self {
Self { ca_pem, key_pem }
}
/// Issue a client leaf carrying `common_name`, returning `(certificate,
/// private key)` as PEM.
///
/// `clientAuth` and nothing else: this certificate authenticates a
/// principal to the store and must not be usable to *serve* anything.
///
/// # Errors
/// When the authority's own PEM will not parse, or the leaf will not sign.
fn mint_leaf(&self, common_name: &str) -> Result<(String, String)> {
let issuer_key = rcgen::KeyPair::from_pem(&self.key_pem)
.context("the agent authority's key is not a PEM key that can sign")?;
let issuer = rcgen::Issuer::from_ca_cert_pem(&self.ca_pem, issuer_key)
.context("the agent authority is not a PEM certificate that can issue")?;
let (not_before, not_after) = validity(SystemTime::now(), LEAF_LIFETIME)?;
let mut params = rcgen::CertificateParams::default();
params.distinguished_name = rcgen::DistinguishedName::new();
params
.distinguished_name
.push(rcgen::DnType::CommonName, common_name);
params.is_ca = rcgen::IsCa::NoCa;
params.use_authority_key_identifier_extension = true;
params.key_usages = vec![
rcgen::KeyUsagePurpose::DigitalSignature,
rcgen::KeyUsagePurpose::KeyEncipherment,
];
params.extended_key_usages = vec![rcgen::ExtendedKeyUsagePurpose::ClientAuth];
params.not_before = not_before;
params.not_after = not_after;
let leaf_key = rcgen::KeyPair::generate().context("generating the leaf's key")?;
let cert = params
.signed_by(&leaf_key, &issuer)
.with_context(|| format!("signing a leaf for {common_name}"))?;
Ok((cert.pem(), leaf_key.serialize_pem()))
}
}
/// The validity window of a leaf minted at `now`, backdated by [`CLOCK_SKEW`].
///
/// A free function taking `now` rather than reading the clock itself, so the
/// arithmetic — the part that can be wrong by a factor of sixty — is testable
/// without waiting ninety days.
/// Where agent leaves are issued: `(mount, role)`, read from `get`.
///
/// # Errors
/// When the system clock is before the unix epoch, or so far past it that the
/// window will not fit a timestamp.
fn validity(now: SystemTime, lifetime: Duration) -> Result<(OffsetDateTime, OffsetDateTime)> {
let secs = now
.duration_since(UNIX_EPOCH)
.context("the system clock is before the unix epoch")?
.as_secs();
let secs = i64::try_from(secs).context("the system clock is past what a timestamp holds")?;
let skew = i64::try_from(CLOCK_SKEW.as_secs()).expect("a five-minute constant fits an i64");
let life = i64::try_from(lifetime.as_secs()).context("the leaf lifetime does not fit")?;
/// Naming the first of the two variables that is unset or empty.
fn agent_pki(get: impl Fn(&str) -> Option<String>) -> Result<(String, String)> {
let required = |var: &str| -> Result<String> {
get(var)
.filter(|v| !v.is_empty())
.with_context(|| format!("{var} is unset or empty, so no agent leaf can be issued"))
};
Ok((
required(ENV_AGENT_PKI_MOUNT)?,
required(ENV_AGENT_PKI_ROLE)?,
))
}
let not_before = OffsetDateTime::from_unix_timestamp(secs - skew)
.context("the backdated start is not a representable time")?;
let not_after = OffsetDateTime::from_unix_timestamp(secs + life)
.context("the expiry is not a representable time")?;
Ok((not_before, not_after))
/// What the cert-auth role for `agent` is written from.
struct RoleInputs<'a> {
/// Role name, policy name and the common name the role matches: one string.
name: String,
/// The authority the role pins: the one that signed this very leaf.
ca: &'a str,
/// The policy document the role attaches.
policy: String,
}
fn role_inputs<'a>(agent: &str, credential: &'a mtls::Credential) -> Result<RoleInputs<'a>> {
Ok(RoleInputs {
name: policy::agent_object_name(agent)?,
ca: &credential.ca,
policy: policy::render_agent(agent)?,
})
}
/// How many bytes of kernel randomness a queue secret is before encoding.
@ -228,49 +109,46 @@ fn generate_queue_secret() -> Result<String> {
/// Give `agent` an identity at the store, and prove it works.
///
/// Five store writes' worth of agreement, then the login that checks it:
/// Five store calls' worth of agreement, then the login that checks it:
///
/// 1. mint a leaf whose common name is [`policy::agent_object_name`];
/// 1. have the agent PKI role issue a leaf whose common name is
/// [`policy::agent_object_name`]; the store generates its key;
/// 2. publish it at [`mtls::identity_path`], where the agent's hive collects
/// it under the hive's own certificate;
/// 3. publish a queue secret at [`queue::agent_queue_path`] — the agent's own
/// identity at the swarm queue, minted here so that the credential an agent
/// presents names *it* rather than its hive;
/// 4. write the ACL document [`policy::render_agent_with_queue`] renders —
/// read on this one agent's paths, plus the hive-shared queue credential
/// every agent container on `hive` already receives out of band;
/// 5. write the cert-auth role that ties the three together.
/// 4. write the ACL document [`policy::render_agent`] renders — read on this
/// one agent's paths and nothing else;
/// 5. write the cert-auth role that ties the three together, pinning the CA
/// the store named as this leaf's issuer.
///
/// Policy before role, for the reason `read_policy::provision` gives: the role
/// names the policy, so the other order leaves a window in which it points at
/// nothing.
///
/// ⚠️ **Step 3 is idempotent and step 2 is not.** Re-running re-mints the
/// certificate — a fresh leaf the agent picks up on its next boot — but leaves
/// an existing queue secret alone. An agent holds that secret in a live
/// connection, and this function is re-run deliberately against agents that
/// are already running, so replacing it would drop them off the queue.
/// Nothing here rotates one; revoking means deleting the path.
/// ⚠️ **Step 3 is idempotent and steps 1–2 are not.** Re-running issues a
/// fresh leaf, picked up on the agent's next boot, but leaves an existing
/// queue secret alone: this is re-run against running agents, which hold that
/// secret in a live connection. Revoking one means deleting the path.
///
/// # Errors
/// Anything that stops one of those five steps, with the step named. A
/// failure here fails the job node and nothing else — the agent is still
/// created, exactly as capable as every agent is today.
pub async fn mint_and_verify(authority: &Authority, agent: &str, hive: &str) -> Result<()> {
/// created, without a store identity.
pub async fn mint_and_verify(agent: &str, hive: &str) -> Result<()> {
let (mount, pki_role) = agent_pki(|k| std::env::var(k).ok())?;
let name = policy::agent_object_name(agent)?;
let path = mtls::identity_path(agent)?;
let queue_path = queue::agent_queue_path(agent)?;
let (cert, key) = authority.mint_leaf(&name)?;
let credential = mtls::Credential {
cert,
key,
ca: authority.ca_pem.clone(),
};
let store = crate::store::connect()
.await
.context("logging in to the swarm secret store")?;
let credential = store
.issue_client_certificate(&mount, &pki_role, &name)
.await
.with_context(|| format!("issuing {name}'s certificate from {mount}/issue/{pki_role}"))?;
store
.write(&path, &credential)
.await
@ -319,32 +197,39 @@ pub async fn mint_and_verify(authority: &Authority, agent: &str, hive: &str) ->
}
let queue_credential = wanted;
let inputs = role_inputs(agent, &credential)?;
store
.write_policy(&name, &policy::render_agent_with_queue(agent, hive)?)
.write_policy(&inputs.name, &inputs.policy)
.await
.with_context(|| format!("writing the read policy {name}"))?;
.with_context(|| format!("writing the read policy {}", inputs.name))?;
store
.write_cert_role(DEFAULT_CERT_MOUNT, &name, &authority.ca_pem, &name, &name)
.write_cert_role(
DEFAULT_CERT_MOUNT,
&inputs.name,
inputs.ca,
&inputs.name,
&inputs.name,
)
.await
.with_context(|| format!("writing the cert-auth role {name}"))?;
.with_context(|| format!("writing the cert-auth role {}", inputs.name))?;
tracing::info!(agent, %path, role = %name, "agent store identity published");
read_back_as_agent(&credential, &name, &path, &queue_path, &queue_credential).await?;
tracing::info!(
agent,
role = %name,
"agent store identity verified: the minted leaf logged in and read both its own paths"
"agent store identity verified: the issued leaf logged in and read both its own paths"
);
Ok(())
}
/// The consumer of everything [`mint_and_verify`] wrote: log in **as the
/// agent**, with the leaf just minted, and read back both paths just
/// agent**, with the leaf just issued, and read back both paths just
/// published.
///
/// The address and the store's CA come from this process's own `BAO_*`
/// environment; the *identity* deliberately does not — see
/// [`SecretStore::connect_with_identity`]. The freshly minted private key
/// [`SecretStore::connect_with_identity`]. The freshly issued private key
/// never touches a filesystem.
///
/// Both paths, not just the certificate's, for the reason this function
@ -377,7 +262,7 @@ async fn read_back_as_agent(
SecretStore::connect_with_identity(&settings, &identity, role, DEFAULT_CERT_MOUNT)
.await
.with_context(|| {
format!("logging in to the store as {role} with the leaf just minted")
format!("logging in to the store as {role} with the leaf just issued")
})?;
let read_back: mtls::Credential = as_agent
.read(path)
@ -407,92 +292,10 @@ async fn read_back_as_agent(
#[cfg(test)]
mod tests {
use super::{
Authority, CLOCK_SKEW, LEAF_LIFETIME, QUEUE_SECRET_BYTES, generate_queue_secret, validity,
ENV_AGENT_PKI_MOUNT, ENV_AGENT_PKI_ROLE, QUEUE_SECRET_BYTES, agent_pki,
generate_queue_secret, role_inputs,
};
use std::time::{Duration, SystemTime, UNIX_EPOCH};
/// A throwaway CA, minted in-process so no test needs a fixture file.
fn test_authority() -> Authority {
let key = rcgen::KeyPair::generate().expect("a key generates");
let mut params = rcgen::CertificateParams::default();
params.is_ca = rcgen::IsCa::Ca(rcgen::BasicConstraints::Constrained(0));
params
.distinguished_name
.push(rcgen::DnType::CommonName, "swarm-agent-ca");
let ca = params.self_signed(&key).expect("the CA self-signs");
Authority::from_pem(ca.pem(), key.serialize_pem())
}
#[test]
fn the_window_is_backdated_by_the_skew_and_as_long_as_the_lifetime() {
// The arithmetic that is wrong by a factor of sixty if a unit slips.
let now = UNIX_EPOCH + Duration::from_secs(1_700_000_000);
let (before, after) = validity(now, LEAF_LIFETIME).expect("a plain instant is fine");
assert_eq!(before.unix_timestamp(), 1_700_000_000 - 300);
assert_eq!(after.unix_timestamp(), 1_700_000_000 + 90 * 24 * 60 * 60);
assert_eq!(CLOCK_SKEW, Duration::from_mins(5));
}
#[test]
fn a_minted_leaf_starts_valid_and_expires() {
let now = i64::try_from(
SystemTime::now()
.duration_since(UNIX_EPOCH)
.expect("the test host's clock is after 1970")
.as_secs(),
)
.expect("and before the end of time");
let (before, after) = validity(SystemTime::now(), LEAF_LIFETIME).expect("now is fine");
assert!(
before.unix_timestamp() < now,
"a leaf usable only in the future is unusable"
);
assert!(
after.unix_timestamp() > now,
"a leaf that has already expired authenticates nothing"
);
// The rule `docs/swarm/credentials.md` states: no credential's
// renewal strategy may read NONE, which starts with it having an end.
// A decade-long leaf is that rule broken in a way review would miss.
assert!(after.unix_timestamp() - now < 3653 * 24 * 60 * 60);
}
/// The four-strings agreement `mint_and_verify` rests on, checked on the
/// one of the four this process controls directly: the certificate really
/// does carry the common name the cert-auth role will be told to match.
#[test]
fn the_leaf_carries_the_agents_object_name_as_its_common_name() {
let name = swarm_secret_client::policy::agent_object_name("atlas").expect("legal");
assert_eq!(name, "hive-agent-atlas");
let (cert, key) = test_authority().mint_leaf(&name).expect("the leaf signs");
assert!(cert.contains("BEGIN CERTIFICATE"), "a PEM certificate");
assert!(key.contains("PRIVATE KEY"), "a PEM key");
// Searched in the SIGNED DER rather than asserted on the params we
// built: the claim is that the name reached the bytes a verifier
// reads. A byte search rather than an X.509 parse because the whole
// crate would otherwise gain a parser dependency for one assertion —
// the name is a UTF8String in the subject DN, so it appears verbatim.
let body: String = cert
.lines()
.filter(|l| !l.starts_with("-----"))
.collect::<Vec<_>>()
.join("");
let der = base64::Engine::decode(&base64::engine::general_purpose::STANDARD, body)
.expect("the PEM body is base64");
assert!(
der.windows(name.len()).any(|w| w == name.as_bytes()),
"the common name must be inside the signed certificate"
);
// And the pair is a usable client identity — the first thing
// `read_back_as_agent` does with it, in exactly this shape.
let mut identity = cert.into_bytes();
identity.push(b'\n');
identity.extend_from_slice(key.as_bytes());
reqwest::Identity::from_pem(&identity).expect("the leaf and its key form a TLS identity");
}
use swarm_secret_client::{mtls, policy};
/// The alphabet claim the token format rests on: the secret is carried in
/// a composite the verifying end splits on `.`, so a secret that could
@ -516,43 +319,65 @@ mod tests {
assert!(!a.contains('='), "{a}");
}
/// A misconfiguration that would otherwise look like "not configured":
/// half an authority has to be an error, not a silent `None`.
///
/// SAFETY: single-threaded mutation of two env vars no other test in this
/// crate reads, removed again before returning.
fn both(k: &str) -> Option<String> {
match k {
ENV_AGENT_PKI_MOUNT => Some("pki-agents".to_owned()),
ENV_AGENT_PKI_ROLE => Some("swarm-agent".to_owned()),
_ => None,
}
}
#[test]
fn half_an_authority_is_an_error_and_neither_half_is_absence() {
unsafe {
std::env::remove_var(super::ENV_AGENT_CA);
std::env::remove_var(super::ENV_AGENT_CA_KEY);
}
assert!(
Authority::from_env()
.expect("neither set is a supported shape")
.is_none(),
"a controller with no authority configured is not an error"
fn the_issue_path_comes_from_the_two_variables_the_module_sets() {
assert_eq!(
agent_pki(both).expect("both set"),
("pki-agents".to_owned(), "swarm-agent".to_owned())
);
}
unsafe { std::env::set_var(super::ENV_AGENT_CA, "/nonexistent/ca.pem") }
// `.err().expect(..)` rather than `expect_err`: the `Ok` half is an
// `Authority`, which deliberately has no `Debug` (it holds a key).
let e = Authority::from_env()
.err()
.expect("a certificate with no key is half an authority");
assert!(format!("{e:#}").contains(super::ENV_AGENT_CA_KEY), "{e:#}");
unsafe {
std::env::remove_var(super::ENV_AGENT_CA);
std::env::set_var(super::ENV_AGENT_CA_KEY, "/nonexistent/ca-key.pem");
#[test]
fn a_missing_or_empty_pki_variable_is_named() {
for var in [ENV_AGENT_PKI_MOUNT, ENV_AGENT_PKI_ROLE] {
let unset = agent_pki(|k| if k == var { None } else { both(k) })
.expect_err("one variable is unset");
assert!(format!("{unset:#}").contains(var), "{unset:#}");
let empty = agent_pki(|k| {
if k == var {
Some(String::new())
} else {
both(k)
}
})
.expect_err("one variable is empty");
assert!(format!("{empty:#}").contains(var), "{empty:#}");
}
// `.err().expect(..)` rather than `expect_err`: the `Ok` half is an
// `Authority`, which deliberately has no `Debug` (it holds a key).
let e = Authority::from_env()
.err()
.expect("a key with no certificate is the other half");
assert!(format!("{e:#}").contains(super::ENV_AGENT_CA), "{e:#}");
}
unsafe { std::env::remove_var(super::ENV_AGENT_CA_KEY) }
/// Three of the four strings that must agree, checked where this process
/// sets them: role, policy and matched CN are one name; the pinned CA is
/// the issuer the store reported for this leaf; the policy is the agent's
/// own single stanza.
#[test]
fn the_role_pins_the_leafs_own_issuer_under_the_agents_name() {
let credential = mtls::Credential {
cert: "LEAF".to_owned(),
key: "KEY".to_owned(),
ca: "AGENT-CA".to_owned(),
};
let inputs = role_inputs("atlas", &credential).expect("legal");
assert_eq!(inputs.name, "hive-agent-atlas");
assert_eq!(
inputs.name,
policy::agent_object_name("atlas").expect("legal"),
"the CN the leaf is issued for"
);
assert_eq!(inputs.ca, "AGENT-CA");
assert_eq!(inputs.policy, policy::render_agent("atlas").expect("legal"));
assert!(!inputs.policy.contains("swarm/hives/"), "{}", inputs.policy);
// The control: another agent's inputs differ in both name and grant.
let other = role_inputs("argus", &credential).expect("legal");
assert_ne!(other.name, inputs.name);
assert!(!other.policy.contains("agents/atlas/"), "{}", other.policy);
}
}

87
swarm-controller/src/main.rs
View file

@ -101,9 +101,9 @@ enum SwarmNodeKind {
/// report success on a write.
///
/// Carries the hive for a different reason than `TriggerDeploy` does:
/// not as an address, but because an agent's ACL document grants read on
/// its hive's shared queue credential, so the document cannot be rendered
/// without knowing which hive the agent belongs to.
/// not as an address, but because the agent's queue credential names the
/// hive it may take subjects on, so it cannot be written without knowing
/// which hive the agent belongs to.
MintAgentIdentity { hive: String, agent: String },
/// Make sure `agent` holds a live forge access token in the swarm secret
/// store, minting one with the forge's admin API when it does not. See
@ -209,11 +209,6 @@ struct WorkerDeps {
/// connection living there is an accident of construction order, not a
/// claim that events are a kind of status.
queue: Option<async_nats::Client>,
/// The authority agent client leaves are issued from, loaded once at
/// startup because it holds a private key and a per-node re-read would be
/// a per-node chance to read one. `None` on a host the operator has not
/// given an authority — see `agent_identity::Authority::from_env`.
agent_ca: Option<std::sync::Arc<agent_identity::Authority>>,
/// The wanted-state writer, for nodes that declare what a hive should
/// converge an agent to. Shares the status reader's queue connection —
/// see `wanted_writer`. `None` exactly when no swarm queue is configured
@ -319,9 +314,7 @@ async fn run_swarm_node(
Err(e) => Outcome::Failed(format!("{e:#}")),
},
},
SwarmNodeKind::MintAgentIdentity { hive, agent } => {
mint_identity(deps.agent_ca.as_deref(), &agent, &hive).await
}
SwarmNodeKind::MintAgentIdentity { hive, agent } => mint_identity(&agent, &hive).await,
SwarmNodeKind::MintAgentForgeToken { agent } => mint_forge_token(deps.forge, &agent).await,
SwarmNodeKind::MintAgentMatrixAccount { agent } => {
mint_matrix_account(deps.matrix_homeserver.as_deref(), &agent).await
@ -347,22 +340,10 @@ async fn run_swarm_node(
/// The `MintAgentIdentity` arm, lifted out so `run_swarm_node` stays under
/// `clippy::too_many_lines`.
async fn mint_identity(
authority: Option<&agent_identity::Authority>,
agent: &str,
hive: &str,
) -> hive_jobq::scheduler::Outcome {
async fn mint_identity(agent: &str, hive: &str) -> hive_jobq::scheduler::Outcome {
use hive_jobq::scheduler::Outcome;
let Some(authority) = authority else {
return Outcome::Failed(
"no agent certificate authority is configured on this host \
(SWARM_CONTROLLER_AGENT_CA_FILE / SWARM_CONTROLLER_AGENT_CA_KEY_FILE unset), \
so this agent has no identity at the swarm secret store"
.to_owned(),
);
};
match agent_identity::mint_and_verify(authority, agent, hive).await {
match agent_identity::mint_and_verify(agent, hive).await {
Ok(()) => Outcome::Done,
Err(e) => Outcome::Failed(format!("{e:#}")),
}
@ -1483,28 +1464,6 @@ fn name_verdict(
}
}
/// The authority agent leaves are issued from, or `None` on a host that was
/// given none.
///
/// Same "log and carry on" shape as `main`'s other optional wiring: a
/// controller with no agent authority still serves everything else, and
/// `MintAgentIdentity` fails with a named reason rather than this process
/// refusing to start. The `Err` arm is worth its own warning — half an
/// authority, or a file that will not read, is a host that looks configured
/// and mints nothing.
fn load_agent_authority() -> Option<Arc<agent_identity::Authority>> {
match agent_identity::Authority::from_env() {
Ok(authority) => authority.map(Arc::new),
Err(e) => {
tracing::warn!(
error = %format!("{e:#}"),
"agent certificate authority unusable; agents get no store identity here"
);
None
}
}
}
/// The sub-DAG one agent creation is: the nodes, and the edges between them.
///
/// A function rather than a closure inside [`create_agent`] so the endpoint's
@ -1589,9 +1548,9 @@ fn declare_agent_job(
//
// `after_any` on the mint, not `after_ok`: a hive cannot pass down a
// certificate the swarm has not published, so the deploy must not
// overtake the mint — but a host with no authority configured must still
// create agents exactly as it does today. `after_ok` there would turn an
// unconfigured option into an agent nobody runs.
// overtake the mint — but a host whose store or agent PKI is not set up
// must still create agents. `after_ok` there would turn a store outage
// into an agent nobody runs.
//
// The forge-token mint gets `after_any` for the same reason: a host with
// no forge or no store must still create agents; only that node fails,
@ -2300,7 +2259,6 @@ async fn main() -> Result<()> {
auth: auth.clone(),
forge: forge_client.clone(),
queue: status.as_ref().map(|s| s.queue_client()),
agent_ca: load_agent_authority(),
wanted: wanted_writer(status.as_ref()),
matrix_homeserver: configured_matrix_homeserver(),
};
@ -2972,7 +2930,6 @@ mod tests {
auth: None,
forge: None,
queue: None,
agent_ca: None,
wanted: None,
matrix_homeserver: None,
};
@ -3027,7 +2984,6 @@ mod tests {
auth: None,
forge: None,
queue: None,
agent_ca: None,
wanted: None,
matrix_homeserver: None,
};
@ -3052,16 +3008,16 @@ mod tests {
}
/// Third sibling of the two above, and the same deliberate caveat: with
/// no authority configured this reaches only the
/// graceful-absence-is-failure branch. The happy path is a live store
/// and a real login, which is exactly why it is `mint_and_verify`'s own
/// job to prove it at agent-creation time rather than a unit test's.
/// no agent PKI named this reaches only the graceful-absence-is-failure
/// branch, before any network call. The happy path is a live store and a
/// real login, which is exactly why it is `mint_and_verify`'s own job to
/// prove it at agent-creation time rather than a unit test's.
///
/// What this does pin is the degrade: a host that was never given an
/// authority fails this one node with a reason that names the two
/// variables, and creates the agent anyway.
/// What this does pin is the degrade: a host whose environment does not
/// name the agent PKI fails this one node with a reason that names the
/// variable, and creates the agent anyway.
#[tokio::test]
async fn mint_agent_identity_node_runs_end_to_end_and_fails_without_an_authority() {
async fn mint_agent_identity_node_runs_end_to_end_and_fails_without_the_agent_pki_named() {
let mut sched = hive_jobq::scheduler::Scheduler::new(
hive_jobq::Graph::new(),
hive_jobq::resources::ResourceTable::new(),
@ -3082,7 +3038,6 @@ mod tests {
auth: None,
forge: None,
queue: None,
agent_ca: None,
wanted: None,
matrix_homeserver: None,
};
@ -3101,9 +3056,8 @@ mod tests {
assert_eq!(node.state, hive_jobq::State::Failed);
let error = node.error.as_deref().unwrap_or_default();
assert!(
error.contains(crate::agent_identity::ENV_AGENT_CA)
&& error.contains(crate::agent_identity::ENV_AGENT_CA_KEY),
"the reason must name both variables an operator has to set, got {error:?}"
error.contains(crate::agent_identity::ENV_AGENT_PKI_MOUNT),
"the reason must name the variable an operator has to set, got {error:?}"
);
}
@ -3135,7 +3089,6 @@ mod tests {
auth: None,
forge: None,
queue: None,
agent_ca: None,
wanted: None,
matrix_homeserver: None,
};
@ -3304,7 +3257,7 @@ mod tests {
.expect("the deploy waits for the mint");
assert!(
when.accepts(hive_jobq::TerminalState::Failed),
"an unconfigured authority must not cancel the deploy; this edge \
"a failed or unconfigured agent PKI issue must not cancel the deploy; this edge \
has to be `after_any`, not `after_ok`"
);
}