vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

stop writing and attaching per-agent and per-hive bao policies

Browse source
This commit is contained in:
atlas 2026-10-26 23:02:08 +01:00
commit 139007e656
4 changed files with 253 additions and 419 deletions
Download patch file Download diff file Expand all files Collapse all files

15
swarm-controller/src/agent_entity.rs
View file

@ -13,11 +13,12 @@
//! is ever moved. Any other arrangement is a [`Plan::Conflict`]: nothing is
//! written and an operator decides.
//!
//! Everything else here is best effort. An agent's own policy still grants it
//! its paths, so a store that refuses these calls costs the shared policy and
//! nothing else: the failure is logged at `warn` and the caller goes on. The
//! grants are the `identity/` stanzas of `swarm-bao.nix`'s
//! `controllerPolicyText`, and a controller can start before they deploy.
//! Everything else here is best effort: a failure is logged at `warn` and the
//! caller goes on. An agent whose entity is not named after it reads nothing,
//! because the shared policy is the only one its role attaches; at mint the
//! read-back fails, and on a start pass its next reads are refused.
//! The grants are the `identity/` stanzas of `swarm-bao.nix`'s
//! `controllerPolicyText`.
use anyhow::{Context, Result, bail};
use swarm_secret_client::{
@ -97,7 +98,7 @@ pub async fn ensure_agents(store: &SecretStore, agents: &[String]) {
Err(e) => {
tracing::warn!(
error = %e,
"agent entities are not managed: the cert auth mount's accessor could not be read; agents keep their own policy"
"agent entities are not managed: the cert auth mount's accessor could not be read; an agent whose entity is not named after it cannot read its secrets"
);
return;
}
@ -173,7 +174,7 @@ async fn ensure_one(store: &SecretStore, accessor: &str, agent: &str) -> Result<
agent,
alias,
entity_name,
"the agent's alias is on an entity that is not the agent's; nothing was changed and the agent keeps its own policy"
"the agent's alias is on an entity that is not the agent's; nothing was changed and the agent cannot read its secrets until an operator resolves it"
);
}
}

76
swarm-controller/src/agent_identity.rs
View file

@ -17,10 +17,10 @@
//! not derived from the first, so either renews without reference to the other.
//!
//! Four separate strings have to agree before an agent can authenticate: the
//! policy's name, the cert-auth role's name, the certificate's common name,
//! and the authority the role pins. That is the kind of agreement that holds
//! in review and fails in production — a mismatch is a 403 naming none of the
//! four — so [`mint_and_verify`] does not finish on a write. See
//! identity entity's name, the cert-auth role's name, the certificate's common
//! name, and the authority the role pins. That is the kind of agreement that
//! holds in review and fails in production — a mismatch is a 403 naming none
//! of the four — so [`mint_and_verify`] does not finish on a write. See
//! [`read_back_as_agent`].
//!
//! The authority is the store's own agent CA, generated inside its agent PKI
@ -62,15 +62,12 @@ fn agent_pki(get: impl Fn(&str) -> Option<String>) -> Result<(String, String)> {
/// What the cert-auth role for `agent` is written from.
struct RoleInputs<'a> {
/// Role name, policy name and the common name the role matches: one string.
/// Role name and the common name the role matches: one string.
name: String,
/// The authority the role pins: the one that signed this very leaf.
ca: &'a str,
/// The policy document the role attaches.
policy: String,
/// The policies the role attaches by name: the agent's own, and the one every
/// agent shares.
attached: [String; 2],
/// The policies the role attaches by name: the one every agent shares.
attached: Vec<String>,
}
fn role_inputs<'a>(agent: &str, credential: &'a mtls::Credential) -> Result<RoleInputs<'a>> {
@ -78,8 +75,7 @@ fn role_inputs<'a>(agent: &str, credential: &'a mtls::Credential) -> Result<Role
Ok(RoleInputs {
name: name.clone(),
ca: &credential.ca,
policy: policy::render_agent(agent)?,
attached: [name, policy::SHARED_AGENT_POLICY.to_owned()],
attached: vec![policy::SHARED_AGENT_POLICY.to_owned()],
})
}
@ -115,7 +111,7 @@ pub(crate) fn generate_queue_secret() -> Result<String> {
/// Give `agent` an identity at the store, and prove it works.
///
/// Five store calls' worth of agreement, then the login that checks it:
/// Six store calls' worth of agreement, then the login that checks it:
///
/// 1. have the agent PKI role issue a leaf whose common name is
/// [`policy::agent_object_name`]; the store generates its key;
@ -124,14 +120,13 @@ pub(crate) fn generate_queue_secret() -> Result<String> {
/// 3. publish a queue secret at [`queue::agent_queue_path`] — the agent's own
/// identity at the swarm queue, minted here so that the credential an agent
/// presents names *it* rather than its hive;
/// 4. write the ACL document [`policy::render_agent`] renders — read and list on
/// this one agent's paths — and the shared [`policy::render_shared_agent`];
/// 5. write the cert-auth role that ties the three together, attaching both
/// policies and pinning the CA the store named as this leaf's issuer.
/// 4. write the shared ACL document [`policy::render_shared_agent`] renders;
/// 5. write the cert-auth role that ties the three together, attaching that
/// policy and pinning the CA the store named as this leaf's issuer;
/// 6. name the agent's identity entity after the agent ([`crate::agent_entity`]),
/// which the policy reads.
///
/// Policy before role, for the reason `read_policy::provision` gives: the role
/// names the policy, so the other order leaves a window in which it points at
/// nothing.
/// Policy before role, for the reason `read_policy::provision` gives.
///
/// ⚠️ **Step 3 is idempotent and steps 1–2 are not.** Re-running issues a
/// fresh leaf, picked up on the agent's next boot, but leaves an existing
@ -140,9 +135,10 @@ pub(crate) fn generate_queue_secret() -> Result<String> {
/// replacing one by age is [`crate::agent_renewal`]'s.
///
/// # Errors
/// Anything that stops one of those five steps, with the step named. A
/// failure here fails the job node and nothing else — the agent is still
/// created, without a store identity.
/// Anything that stops steps 1–5, with the step named, or the read-back login,
/// which is where a step 6 that did not take shows up. A failure here fails the
/// job node and nothing else — the agent is still created, without a store
/// identity.
pub async fn mint_and_verify(agent: &str) -> Result<()> {
let (mount, pki_role) = agent_pki(|k| std::env::var(k).ok())?;
let name = policy::agent_object_name(agent)?;
@ -201,10 +197,6 @@ pub async fn mint_and_verify(agent: &str) -> Result<()> {
let queue_credential = wanted;
let inputs = role_inputs(agent, &credential)?;
store
.write_policy(&inputs.name, &inputs.policy)
.await
.with_context(|| format!("writing the read policy {}", inputs.name))?;
store
.write_policy(policy::SHARED_AGENT_POLICY, &policy::render_shared_agent())
.await
@ -220,7 +212,11 @@ pub async fn mint_and_verify(agent: &str) -> Result<()> {
&inputs.name,
inputs.ca,
&inputs.name,
&inputs.attached.each_ref().map(String::as_str),
&inputs
.attached
.iter()
.map(String::as_str)
.collect::<Vec<_>>(),
)
.await
.with_context(|| format!("writing the cert-auth role {}", inputs.name))?;
@ -297,7 +293,7 @@ pub async fn revoke_queue_credential(agent: &str) -> Result<()> {
///
/// Both paths, not just the certificate's, for the reason this function
/// exists at all: the queue credential is readable only because it sits inside
/// the stanza [`policy::render_agent`] already grants, and a policy that
/// the stanza [`policy::render_shared_agent`] grants, and a policy that
/// drifted from that path would otherwise fail at the agent's first connection
/// — far from here, as a denial naming neither the policy nor the path.
///
@ -450,9 +446,9 @@ mod tests {
}
/// Three of the four strings that must agree, checked where this process
/// sets them: role, policy and matched CN are one name; the pinned CA is
/// the issuer the store reported for this leaf; the policy is the agent's
/// own single stanza.
/// sets them: role and matched CN are one name; the pinned CA is the issuer
/// the store reported for this leaf; the role attaches the shared policy
/// and no policy of the agent's own.
#[test]
fn the_role_pins_the_leafs_own_issuer_under_the_agents_name() {
let credential = mtls::Credential {
@ -468,17 +464,21 @@ mod tests {
"the CN the leaf is issued for"
);
assert_eq!(inputs.ca, "AGENT-CA");
// Only the shared policy: a policy named for the agent is not attached.
assert_eq!(
inputs.attached,
["hive-agent-atlas", policy::SHARED_AGENT_POLICY],
"the role attaches the agent's own policy and the shared one"
[policy::SHARED_AGENT_POLICY],
"the role attaches the shared policy and nothing else"
);
assert!(
!inputs.attached.contains(&inputs.name),
"the role must not attach a policy named for the agent"
);
assert_eq!(inputs.policy, policy::render_agent("atlas").expect("legal"));
assert!(!inputs.policy.contains("swarm/hives/"), "{}", inputs.policy);
// The control: another agent's inputs differ in both name and grant.
// The control: another agent's inputs differ in name, and attach the
// same shared policy, so the check above can see a policy that is there.
let other = role_inputs("argus", &credential).expect("legal");
assert_ne!(other.name, inputs.name);
assert!(!other.policy.contains("agents/atlas/"), "{}", other.policy);
assert_eq!(other.attached, inputs.attached);
}
}

4
swarm-controller/src/main.rs
View file

@ -217,8 +217,8 @@ async fn main() -> Result<()> {
// abandoned, since the two of them boot together.
let hive_names: Vec<String> = hives.iter().map(|h| h.name.clone()).collect();
read_policy::ensure_hive_access(hive_names).await;
// Agents keep the policy they were minted with until this rewrites it, so
// what `policy::render_agent` grants today reaches them only from here.
// The one policy every agent's role attaches, and the entity each agent's
// login is read under, are checked here at every start.
read_policy::ensure_agent_policies().await;
let state = AppState {

179
swarm-controller/src/read_policy.rs
View file

@ -14,18 +14,16 @@
//! take down a daemon that serves everything else, and a store that answers is
//! answering the same way next time, so the next start is the retry for those.
//!
//! The document names the hive it is written for: one stanza is the same
//! everywhere, the other is scoped to the reader's own name
//! ([`swarm_secret_client::policy`] explains the asymmetry). The same rule is
//! also written once for all hives, [`policy::SHARED_HIVE_POLICY`], with the
//! reader's own name taken from its login's alias on the cert mount instead of
//! written into the text; a hive's role attaches both. The hive list is loaded
//! once because a config change means a redeploy.
//! The rule is written once for all hives, [`policy::SHARED_HIVE_POLICY`]: one
//! stanza is the same everywhere, the other is scoped to the reader's own name
//! ([`swarm_secret_client::policy`] explains the asymmetry), taken from its
//! login's alias on the cert mount instead of written into the text. A hive's
//! role attaches that policy and no other. The hive list is loaded once because
//! a config change means a redeploy.
//!
//! Every agent's policy is rewritten by the same kind of pass
//! ([`ensure_agent_policies`]), so a change to `policy::render_agent` reaches
//! agents minted before it. Its roster is the store's `hive-agent-*` cert-auth
//! roles.
//! The shared agent policy is written by the same kind of pass
//! ([`ensure_agent_policies`]), together with each agent's identity entity.
//! Its roster is the store's `hive-agent-*` cert-auth roles.
use std::{future::Future, time::Duration};
@ -128,10 +126,12 @@ where
None
}
/// One pass: log in once, write every hive's policy, then every hive's role.
/// One pass: log in once, write the shared hive policy, then every hive's role.
///
/// Policy first, because the role names the policy — the other order leaves a
/// window in which the role points at nothing.
/// window in which the role points at nothing. Without the policy the roles are
/// left as they are: a role written to name a policy that does not exist logs
/// its hive in able to read nothing.
///
/// # Errors
/// [`Unreachable`] for the states a later attempt can resolve: a store that is
@ -154,12 +154,9 @@ async fn provision(hives: &[String]) -> Result<(), Unreachable> {
}
Err(e) => return Err(Unreachable(e.to_string())),
};
for hive in hives {
if let Err(e) = write_policy_for(&store, hive).await {
tracing::warn!(hive, error = %format!("{e:#}"), "writing this hive's read policy failed");
}
if !write_shared_policy(&store).await {
return Ok(());
}
let shared = write_shared_policy(&store).await;
let Some(path) = std::env::var_os(ENV_HIVE_CLIENT_CA) else {
tracing::info!("no hive certificate authority configured; hive roles are not managed here");
@ -171,38 +168,25 @@ async fn provision(hives: &[String]) -> Result<(), Unreachable> {
let ca = std::fs::read_to_string(&path)
.map_err(|e| Unreachable(format!("reading the hive authority {path}: {e}")))?;
for hive in hives {
if let Err(e) = write_role(&store, hive, &ca, shared).await {
if let Err(e) = write_role(&store, hive, &ca).await {
tracing::warn!(hive, error = %format!("{e:#}"), "creating this hive's store role failed");
}
}
Ok(())
}
/// One hive's policy: its own name, its own document.
async fn write_policy_for(store: &SecretStore, hive: &str) -> Result<()> {
let name = policy::hive_object_name(hive)?;
store
.write_policy(&name, &policy::render(hive)?)
.await
.with_context(|| format!("writing the read policy {name}"))?;
tracing::info!(hive, policy = %name, "hive read policy in place");
Ok(())
}
/// Write [`policy::SHARED_HIVE_POLICY`], rendered with the cert mount's
/// accessor. Answers whether it is in place: a hive's role attaches it only
/// then.
/// accessor. Answers whether it is in place: hive roles are written only then.
///
/// Best effort. A hive's own policy grants it the same paths, so a store that
/// refuses the accessor read (the controller's policy may not carry it yet) or
/// the write costs the shared policy and nothing else.
/// A store that refuses the accessor read (the controller's policy may not
/// carry it yet) or the write leaves the roles already in the store as they are.
async fn write_shared_policy(store: &SecretStore) -> bool {
let accessor = match store.auth_mount_accessor(DEFAULT_CERT_MOUNT).await {
Ok(accessor) => accessor,
Err(e) => {
tracing::warn!(
error = %e,
"the shared hive policy is not managed: the cert auth mount's accessor could not be read; hives keep their own policy"
"the shared hive policy is not managed: the cert auth mount's accessor could not be read; hive roles are left as they are"
);
return false;
}
@ -210,7 +194,7 @@ async fn write_shared_policy(store: &SecretStore) -> bool {
let document = match policy::render_shared_hives(&accessor) {
Ok(document) => document,
Err(e) => {
tracing::warn!(error = %e, "the shared hive policy is not managed: the cert auth mount's accessor is not usable in a policy; hives keep their own policy");
tracing::warn!(error = %e, "the shared hive policy is not managed: the cert auth mount's accessor is not usable in a policy; hive roles are left as they are");
return false;
}
};
@ -229,28 +213,23 @@ async fn write_shared_policy(store: &SecretStore) -> bool {
tracing::warn!(
policy = policy::SHARED_HIVE_POLICY,
error = %e,
"writing the shared hive policy failed; hives keep their own policy"
"writing the shared hive policy failed; hive roles are left as they are"
);
false
}
}
}
/// The policies `hive`'s role attaches: its own, and the shared one when that
/// is in place.
fn attached_policies(hive: &str, shared: bool) -> Result<Vec<String>, Error> {
let mut attached = vec![policy::hive_object_name(hive)?];
if shared {
attached.push(policy::SHARED_HIVE_POLICY.to_owned());
}
Ok(attached)
/// The policies a hive's role attaches: the shared one, and no policy named for
/// the hive.
fn attached_policies() -> Vec<String> {
vec![policy::SHARED_HIVE_POLICY.to_owned()]
}
/// The role for one hive: its own certificate's subject, its own policy, and
/// the shared one when `shared`.
async fn write_role(store: &SecretStore, hive: &str, ca: &str, shared: bool) -> Result<()> {
/// The role for one hive: its own certificate's subject and the shared policy.
async fn write_role(store: &SecretStore, hive: &str, ca: &str) -> Result<()> {
let name = policy::hive_object_name(hive)?;
let attached = attached_policies(hive, shared)?;
let attached = attached_policies();
store
// The subject is the hive's name: `glue-bao-tls.nix` mints a hive's
// client leaf with its name as the CN, and cert auth matches on that.
@ -267,8 +246,8 @@ async fn write_role(store: &SecretStore, hive: &str, ca: &str, shared: bool) ->
Ok(())
}
/// Rewrite every agent's policy from [`policy::render_agent`], with the retry
/// [`ensure_hive_access`] has and for the same reason.
/// Write the shared agent policy and give every agent its identity entity, with
/// the retry [`ensure_hive_access`] has and for the same reason.
///
/// Returns once the first pass is done. When that pass cannot reach the store,
/// it repeats in the background for up to [`RETRY_WINDOW`].
@ -294,7 +273,7 @@ pub async fn ensure_agent_policies() {
tracing::warn!(
attempts = RETRY_ATTEMPTS,
window = ?RETRY_WINDOW,
"giving up on the agent policy rewrite; agents keep the policy they were minted with until this daemon is restarted"
"giving up on the agent policy rewrite; the shared agent policy and the agents' identity entities are not checked until this daemon is restarted"
);
}
});
@ -302,9 +281,9 @@ pub async fn ensure_agent_policies() {
}
/// One pass: log in once, list the agents, write the shared agent policy and
/// each one's own, and give each its identity entity.
/// give each agent its identity entity.
///
/// Policies and entities only. An agent's cert-auth role and certificate are
/// The policy and entities only. An agent's cert-auth role and certificate are
/// `agent_identity::mint_and_verify`'s, and that reissues the certificate, so
/// it is not what a pass on every start may call.
///
@ -343,46 +322,17 @@ async fn rewrite_agent_policies() -> Result<(), Unreachable> {
"writing the shared agent policy failed"
),
}
let policies = agent_policies(&roles);
let mut written = 0_usize;
for (name, document) in &policies {
match store.write_policy(name, document).await {
Ok(()) => written += 1,
Err(e) => {
tracing::warn!(policy = %name, error = %e, "rewriting this agent's policy failed");
}
}
}
tracing::info!(written, agents = policies.len(), "agent policies rewritten");
crate::agent_entity::ensure_agents(&store, &policy::agents_from_role_names(&roles)).await;
Ok(())
}
/// The policy to write for each agent among `roles`, a listing of cert-auth
/// roles: its name, which is the role's own, and its document.
///
/// A role that is not an agent's — a hive's `hive-<hive>`, a host's — gets
/// nothing: the same `write_policy` call with an agent's document would
/// narrow that principal to one agent's subtree.
fn agent_policies(roles: &[String]) -> Vec<(String, String)> {
policy::agents_from_role_names(roles)
.iter()
.filter_map(|agent| {
Some((
policy::agent_object_name(agent).ok()?,
policy::render_agent(agent).ok()?,
))
})
.collect()
}
#[cfg(test)]
mod tests {
use std::{cell::Cell, time::Duration};
use super::{
RETRY_ATTEMPTS, RETRY_INTERVAL, RETRY_WINDOW, Unreachable, agent_policies,
attached_policies, policy, retry_until_ok,
RETRY_ATTEMPTS, RETRY_INTERVAL, RETRY_WINDOW, Unreachable, attached_policies, policy,
retry_until_ok,
};
#[test]
@ -424,54 +374,15 @@ mod tests {
}
#[test]
fn the_pass_writes_agents_policies_under_their_role_names_and_no_one_elses() {
let roles: Vec<String> = [
"hive-agent-atlas",
"hive-pr1ma",
"swarm-controller",
"hive-agent-argus",
]
.map(str::to_owned)
.to_vec();
let written = agent_policies(&roles);
let names: Vec<&str> = written.iter().map(|(name, _)| name.as_str()).collect();
// The role names an agent's policy by its own name, so a policy written
// under any other name is one the role does not attach.
assert_eq!(names, ["hive-agent-atlas", "hive-agent-argus"]);
assert_eq!(
written[0].1,
policy::render_agent("atlas").expect("legal"),
"the prefix is stripped before rendering, or the document names `agent-atlas`"
fn a_hives_role_attaches_the_shared_policy_and_none_named_for_the_hive() {
let attached = attached_policies();
assert_eq!(attached, [policy::SHARED_HIVE_POLICY]);
let own = policy::hive_object_name("pr1ma").expect("legal");
assert!(
!attached.contains(&own),
"the role must not attach a policy named for the hive"
);
assert_eq!(written[1].1, policy::render_agent("argus").expect("legal"));
}
#[test]
fn a_hives_role_attaches_its_own_policy_and_the_shared_one() {
assert_eq!(
attached_policies("pr1ma", true).expect("legal"),
["hive-pr1ma", policy::SHARED_HIVE_POLICY]
);
}
#[test]
fn without_the_shared_policy_the_role_keeps_the_hives_own() {
assert_eq!(
attached_policies("pr1ma", false).expect("legal"),
["hive-pr1ma"]
);
}
#[test]
fn the_shared_text_serves_every_hive_what_its_own_policy_does() {
let shared = policy::render_shared_hives("auth_cert_0a1b2c3d").expect("legal accessor");
let template = "{{identity.entity.aliases.auth_cert_0a1b2c3d.name}}";
for hive in ["pr1ma", "beta", "gamma-2", "delta_3"] {
assert_eq!(
shared.replace(template, hive),
policy::render(hive).expect("legal"),
"{hive}"
);
}
// The control: the same lookup finds a policy that is attached.
assert!(attached.contains(&policy::SHARED_HIVE_POLICY.to_owned()));
}
}