vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

swarm: mint, publish and login-verify an agent's store identity at create

Browse source 
`swarm/agents/<agent>/bao-mtls` did not exist, and neither did any
per-agent identity at the secret store: `policy::agent_object_name`,
`render_agent` and `render_agent_with_queue` had been written and never
called outside their own tests. An agent's only "per-agent" secret today
is read under the HIVE's certificate, through a wide grant on
`swarm/agents/*` — so "per-agent" was presentational.

The swarm now mints the certificate, so no hive ever needs the capability
to mint one. `swarm-controller` is the service that does it: it already
logs in to the store, and its existing grant already covers exactly the
three objects written here (`create/update` on
`secret/data/swarm/agents/*`, `sys/policies/acl/hive-*` and
`auth/cert/certs/hive-*`). No new bao grant, and nothing co-located — a
cert-auth role pins its authority by value, per role, so the controller
issues from its own CA on its own host and pins that CA in the role it
writes. No existing role changes.

The mint node does not report success on a write. After publishing it
connects again, with the leaf it just issued and under the role it just
wrote, and reads the path back — so the policy, the role, the common name
and the leaf are exercised in production on every agent creation. A
certificate this code mints that the role this code writes will not accept
turns the job node red at creation time instead of surfacing later as an
agent container that cannot start.

`TriggerDeploy` gains an `after_any` edge on the mint, not `after_ok`: a
hive cannot pass down a certificate the swarm has not published, but a
host with no authority configured must still create agents exactly as it
does today.

The private key is generated in memory and never written to disk on the
controller — `SecretStore::connect_with_identity` takes the PEM the minter
is already holding, so nothing is written out purely to be logged in with.

Refs #4137
This commit is contained in:
atlas 2026-09-18 15:05:24 +02:00
commit 676c45bc93
10 changed files with 1262 additions and 71 deletions
Download patch file Download diff file Expand all files Collapse all files

301
swarm-controller/src/main.rs
View file

@ -40,6 +40,7 @@ use swarm_authelia_bridge_sock::BridgeResponse;
use utoipa::{OpenApi, ToSchema};
use utoipa_axum::{router::OpenApiRouter, routes};
mod agent_identity;
mod agent_state_stream;
mod agent_status;
mod auth;
@ -94,6 +95,16 @@ enum SwarmNodeKind {
/// swarm routes a deploy message to — it belongs on the node that
/// sends that message, not on this one.
InitAgentConfigRepo { agent: String },
/// Mint `agent`'s own client certificate for the swarm secret store,
/// publish it there, grant it, and log in with it. See
/// `agent_identity::mint_and_verify` — including why this node does not
/// report success on a write.
///
/// Carries the hive for a different reason than `TriggerDeploy` does:
/// not as an address, but because an agent's ACL document grants read on
/// its hive's shared queue credential, so the document cannot be rendered
/// without knowing which hive the agent belongs to.
MintAgentIdentity { hive: String, agent: String },
/// Tell `hive` to rebuild `agent`, by publishing on the swarm's deploy
/// subject. The one node kind whose effect leaves this host.
///
@ -112,6 +123,7 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
SwarmNodeKind::CreateForgeUser { .. } => "create_forge_user".to_owned(),
SwarmNodeKind::AddRepoMember { .. } => "add_repo_member".to_owned(),
SwarmNodeKind::InitAgentConfigRepo { .. } => "init_agent_config_repo".to_owned(),
SwarmNodeKind::MintAgentIdentity { .. } => "mint_agent_identity".to_owned(),
SwarmNodeKind::TriggerDeploy { .. } => "trigger_deploy".to_owned(),
}
}
@ -131,7 +143,8 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
| SwarmNodeKind::InitAgentConfigRepo { agent } => {
serde_json::json!({ "agent": agent })
}
SwarmNodeKind::TriggerDeploy { hive, agent } => {
SwarmNodeKind::TriggerDeploy { hive, agent }
| SwarmNodeKind::MintAgentIdentity { hive, agent } => {
serde_json::json!({ "agent": agent, "hive": hive })
}
}
@ -168,6 +181,11 @@ struct WorkerDeps {
/// connection living there is an accident of construction order, not a
/// claim that events are a kind of status.
queue: Option<async_nats::Client>,
/// The authority agent client leaves are issued from, loaded once at
/// startup because it holds a private key and a per-node re-read would be
/// a per-node chance to read one. `None` on a host the operator has not
/// given an authority — see `agent_identity::Authority::from_env`.
agent_ca: Option<std::sync::Arc<agent_identity::Authority>>,
}
/// Run a claimed node's actual work. Mirrors `hive-c0re/src/job_queue/
@ -254,6 +272,20 @@ async fn run_swarm_node(
Err(e) => Outcome::Failed(format!("{e:#}")),
},
},
SwarmNodeKind::MintAgentIdentity { hive, agent } => match deps.agent_ca {
None => Outcome::Failed(
"no agent certificate authority is configured on this host \
(SWARM_CONTROLLER_AGENT_CA_FILE / SWARM_CONTROLLER_AGENT_CA_KEY_FILE unset), \
so this agent has no identity at the swarm secret store"
.to_owned(),
),
Some(authority) => {
match agent_identity::mint_and_verify(&authority, &agent, &hive).await {
Ok(()) => Outcome::Done,
Err(e) => Outcome::Failed(format!("{e:#}")),
}
}
},
SwarmNodeKind::TriggerDeploy { hive, agent } => match deps.queue {
None => Outcome::Failed(
"no swarm queue is configured on this host, so no hive can be told to deploy"
@ -1194,50 +1226,7 @@ async fn create_agent(
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let ids = sched
.insert_job(None, |b| {
let create_identity = b.node(SwarmNodeKind::CreateIdentity {
agent: agent.clone(),
});
// A second, independent root: a forge user needs neither an
// authelia subject nor an existing repo, so it does not chain
// off `create_identity` (see the doc comment above).
let create_forge_user = b.node(SwarmNodeKind::CreateForgeUser {
agent: agent.clone(),
});
let create_repo = b
.node(SwarmNodeKind::CreateRepo {
agent: agent.clone(),
})
.after_ok(create_identity);
// `AddRepoMember` needs both parents: the repo to add a
// collaborator to, and the forge user to add as one — adding a
// nonexistent user is a Forgejo validation error, not
// an idempotent no-op. `InitAgentConfigRepo` needs only the
// repo — see the doc comment above for why.
let _add_repo_member = b
.node(SwarmNodeKind::AddRepoMember {
agent: agent.clone(),
})
.after_ok(create_repo)
.after_ok(create_forge_user);
let init_config = b
.node(SwarmNodeKind::InitAgentConfigRepo {
agent: agent.clone(),
})
.after_ok(create_repo);
// Last, and specifically after the config repo is seeded: the
// hive deploys by reading that repo, so a deploy asked for any
// earlier would find nothing to build. This is the edge that
// makes creating an agent at swarm level actually put it on a
// hive, rather than leaving a provisioned name nobody runs.
let _trigger_deploy = b
.node(SwarmNodeKind::TriggerDeploy {
hive: hive.clone(),
agent,
})
.after_ok(init_config);
vec![create_identity.guid()]
})
.insert_job(None, |b| declare_agent_job(b, &agent, &hive))
.map_err(|e| {
error_problem(
axum::http::StatusCode::INTERNAL_SERVER_ERROR,
@ -1253,6 +1242,101 @@ async fn create_agent(
}))
}
/// The authority agent leaves are issued from, or `None` on a host that was
/// given none.
///
/// Same "log and carry on" shape as `main`'s other optional wiring: a
/// controller with no agent authority still serves everything else, and
/// `MintAgentIdentity` fails with a named reason rather than this process
/// refusing to start. The `Err` arm is worth its own warning — half an
/// authority, or a file that will not read, is a host that looks configured
/// and mints nothing.
fn load_agent_authority() -> Option<Arc<agent_identity::Authority>> {
match agent_identity::Authority::from_env() {
Ok(authority) => authority.map(Arc::new),
Err(e) => {
tracing::warn!(
error = %format!("{e:#}"),
"agent certificate authority unusable; agents get no store identity here"
);
None
}
}
}
/// The sub-DAG one agent creation is: the nodes, and the edges between them.
///
/// A function rather than a closure inside [`create_agent`] so the endpoint's
/// validation and the graph's shape can each be read without scrolling past
/// the other — and so the one handle the response reports is returned from
/// the place that decides which node it is.
fn declare_agent_job(
b: &hive_jobq::builder::JobBuilder<SwarmNodeKind, SwarmResourceKind>,
agent: &str,
hive: &str,
) -> Vec<hive_jobq::builder::NodeGuid> {
let create_identity = b.node(SwarmNodeKind::CreateIdentity {
agent: agent.to_owned(),
});
// A second, independent root: a forge user needs neither an authelia
// subject nor an existing repo, so it does not chain off
// `create_identity` (see the doc comment above).
let create_forge_user = b.node(SwarmNodeKind::CreateForgeUser {
agent: agent.to_owned(),
});
let create_repo = b
.node(SwarmNodeKind::CreateRepo {
agent: agent.to_owned(),
})
.after_ok(create_identity);
// `AddRepoMember` needs both parents: the repo to add a collaborator to,
// and the forge user to add as one — adding a nonexistent user is a
// Forgejo validation error, not an idempotent no-op.
// `InitAgentConfigRepo` needs only the repo — see the doc comment above
// for why.
let _add_repo_member = b
.node(SwarmNodeKind::AddRepoMember {
agent: agent.to_owned(),
})
.after_ok(create_repo)
.after_ok(create_forge_user);
let init_config = b
.node(SwarmNodeKind::InitAgentConfigRepo {
agent: agent.to_owned(),
})
.after_ok(create_repo);
// The agent's own identity at the swarm secret store, minted and
// published at swarm level so no hive ever needs the capability to mint
// one. After `create_identity` because the certificate names a principal
// the swarm has agreed exists — not because anything in the store reads
// authelia.
let mint_identity = b
.node(SwarmNodeKind::MintAgentIdentity {
hive: hive.to_owned(),
agent: agent.to_owned(),
})
.after_ok(create_identity);
// Last, and specifically after the config repo is seeded: the hive
// deploys by reading that repo, so a deploy asked for any earlier would
// find nothing to build. This is the edge that makes creating an agent at
// swarm level actually put it on a hive, rather than leaving a
// provisioned name nobody runs.
//
// `after_any` on the mint, not `after_ok`: a hive cannot pass down a
// certificate the swarm has not published, so the deploy must not
// overtake the mint — but a host with no authority configured must still
// create agents exactly as it does today. `after_ok` there would turn an
// unconfigured option into an agent nobody runs.
let _trigger_deploy = b
.node(SwarmNodeKind::TriggerDeploy {
hive: hive.to_owned(),
agent: agent.to_owned(),
})
.after_ok(init_config)
.after_any(mint_identity);
vec![create_identity.guid()]
}
/// Query params for `GET /api/jobq/graph` — `?states=` narrows to root
/// groups in the named states, same shape `hive_jobq_wire::parse_states`
/// parses.
@ -1579,6 +1663,7 @@ async fn main() -> Result<()> {
auth: auth.clone(),
forge: forge_client.clone(),
queue: status.as_ref().map(|s| s.queue_client()),
agent_ca: load_agent_authority(),
};
let jobq = Arc::new(Mutex::new(hive_jobq::scheduler::Scheduler::new(
@ -1945,6 +2030,7 @@ mod tests {
auth: None,
forge: None,
queue: None,
agent_ca: None,
};
let runner =
hive_jobq::scheduler::Scheduler::claim_next(&sched, move |id, kind, builder| {
@ -1997,6 +2083,7 @@ mod tests {
auth: None,
forge: None,
queue: None,
agent_ca: None,
};
let runner =
hive_jobq::scheduler::Scheduler::claim_next(&sched, move |id, kind, builder| {
@ -2018,6 +2105,130 @@ mod tests {
);
}
/// Third sibling of the two above, and the same deliberate caveat: with
/// no authority configured this reaches only the
/// graceful-absence-is-failure branch. The happy path is a live store
/// and a real login, which is exactly why it is `mint_and_verify`'s own
/// job to prove it at agent-creation time rather than a unit test's.
///
/// What this does pin is the degrade: a host that was never given an
/// authority fails this one node with a reason that names the two
/// variables, and creates the agent anyway.
#[tokio::test]
async fn mint_agent_identity_node_runs_end_to_end_and_fails_without_an_authority() {
let mut sched = hive_jobq::scheduler::Scheduler::new(
hive_jobq::Graph::new(),
hive_jobq::resources::ResourceTable::new(),
);
let id = sched
.append(
SwarmNodeKind::MintAgentIdentity {
hive: "pr1ma".to_owned(),
agent: "atlas".to_owned(),
},
Vec::new(),
None,
)
.expect("insert");
let sched = std::sync::Arc::new(std::sync::Mutex::new(sched));
let deps = WorkerDeps {
auth: None,
forge: None,
queue: None,
agent_ca: None,
};
let runner =
hive_jobq::scheduler::Scheduler::claim_next(&sched, move |id, kind, builder| {
run_swarm_node(id, kind, builder, deps)
})
.expect("the node just inserted is runnable");
runner
.await
.1
.expect("no growth declared, nothing to reject");
let guard = sched.lock().unwrap();
let node = guard.graph().node(id).expect("node still present");
assert_eq!(node.state, hive_jobq::State::Failed);
let error = node.error.as_deref().unwrap_or_default();
assert!(
error.contains(crate::agent_identity::ENV_AGENT_CA)
&& error.contains(crate::agent_identity::ENV_AGENT_CA_KEY),
"the reason must name both variables an operator has to set, got {error:?}"
);
}
/// The node carries the hive, and the viewer has to see it. The `data`
/// match is an or-pattern on purpose (see its own comment), and this is
/// the assertion that the new variant joined the two-field arm rather
/// than the agent-only one — a viewer silently missing the hive is the
/// failure that comment describes having already happened once.
#[test]
fn a_mint_node_renders_both_the_agent_and_the_hive() {
use hive_jobq_wire::WireNode as _;
let kind = SwarmNodeKind::MintAgentIdentity {
hive: "pr1ma".to_owned(),
agent: "atlas".to_owned(),
};
assert_eq!(kind.label(), "mint_agent_identity");
let data = kind.data(1);
assert_eq!(data["agent"], "atlas");
assert_eq!(data["hive"], "pr1ma");
}
/// The ordering the operator's ruling requires: a hive cannot pass down
/// a certificate the swarm has not published, so the deploy message must
/// not leave before the mint is terminal.
///
/// Asserted on the graph `create_agent` builds, because the edge is one
/// line in a builder closure and its absence changes nothing observable
/// until a real agent boots without an identity.
#[tokio::test]
async fn the_deploy_waits_for_the_mint_and_is_not_cancelled_by_it() {
use hive_jobq_wire::WireNode as _;
let (state, sched) = state_with_roster();
let _queued = super::create_agent(
axum::extract::State(state),
axum::Json(super::CreateAgentRequest {
name: "atlas".to_owned(),
hive: "pr1ma".to_owned(),
}),
)
.await
.expect("a hive in the roster must be accepted");
let guard = sched
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let graph = guard.graph();
let id_of = |label: &str| {
graph
.nodes()
.find(|n| n.payload.label() == label)
.unwrap_or_else(|| panic!("the graph holds a {label} node"))
.id
};
let mint = id_of("mint_agent_identity");
let deploy = graph.node(id_of("trigger_deploy")).expect("just found");
let when = deploy
.deps
.iter()
.find_map(|d| match d {
hive_jobq::Dep::Node { id, when } if *id == mint => Some(*when),
_ => None,
})
.expect("the deploy waits for the mint");
assert!(
when.accepts(hive_jobq::TerminalState::Failed),
"an unconfigured authority must not cancel the deploy; this edge \
has to be `after_any`, not `after_ok`"
);
}
/// The socket must not share a directory with anything else, because
/// the socket is `0666` and the directory is therefore the only access
/// control it has. `/run/hyperhive` in particular holds hive-c0re's