swarm: courier an agent's store identity into its container, and log in with it
`swarm-controller` mints an agent's mTLS leaf at creation and publishes it at `swarm/agents/<agent>/bao-mtls`. Nothing read it back. This adds the hop that carries it the rest of the way, and the in-container consumer that proves the hop works. Host side, `lifecycle::agent_identity` reads the row under *this hive's* own certificate — the hive is a principal the store already knows — and stages the leaf and its key `0600` under a new `agent-identity/<name>` state dir, deliberately outside every bind-mounted tree. Both files go in as systemd credentials rather than binds, the same answer and the same mode reason as the queue secret beside it: the staged key is unreadable to the unprivileged agent user, and the container manager reads a `--load-credential` source as root before re-exposing it under the consuming unit's own `User=`. The agent is never asked to authenticate in order to obtain the thing it authenticates with. Container side, `hive-agent-bao-identity.service` logs in with that certificate and reads the agent's own path back, failing the unit when either step does not succeed. It fails loudly where the hive-side readers degrade quietly, because a refused certificate means an agent that believes it reaches the store and never does — a cause only the login itself can name. The address is the whole switch, no separate `enable`, matching how `queue.nix` and `logs.nix` already gate themselves. A hive with a store forwards `HIVE_AGENT_BAO_ADDR` and every agent on it gets the check; a hive without one forwards nothing and no agent does. That is what keeps the delivery from landing in a container with nothing to read it. The hive can now reach an agent's identity, so hive privilege covers agent privilege. Accepted, not mitigated: the alternative is an agent fetching its own credential with a credential it does not yet have. Refs #4137
This commit is contained in:
atlas
parent
8aafe4eaee
commit
837e658d4a
10 changed files with 716 additions and 7 deletions
306
hive-c0re/src/lifecycle/agent_identity.rs
Normal file
306
hive-c0re/src/lifecycle/agent_identity.rs
Normal file
|
|
@ -0,0 +1,306 @@
|
|||
//! The courier hop: this hive collects an agent's own store identity and
|
||||
//! stages it for delivery into that agent's container.
|
||||
//!
|
||||
//! `swarm-controller` mints the leaf at agent-create and publishes it at
|
||||
//! `swarm/agents/<agent>/bao-mtls`; this reads it back under **the hive's**
|
||||
//! certificate — the hive is a principal the store already knows — and lands
|
||||
//! the two files `systemd-nspawn` forwards as credentials. The agent is never
|
||||
//! asked to authenticate in order to obtain the thing it authenticates with:
|
||||
//! by the time it runs, its identity is already inside it.
|
||||
//!
|
||||
//! **A credential and not a bind mount**, the same answer and the same reason
|
||||
//! as the queue secret beside it in [`super::host_config`]: the files are
|
||||
//! `0600` to `hive-core` and the units that use them run as the unprivileged
|
||||
//! agent user, so a bind would deliver a private key that user cannot open.
|
||||
//! The container manager reads a `--load-credential` source as root and
|
||||
//! re-exposes it under the consuming unit's own `User=`.
|
||||
//!
|
||||
//! 🩸 Nothing here ever formats the credential's fields. The value moves
|
||||
//! store → memory → file and is never an argument to a log macro; the
|
||||
//! [`mtls::Credential`] type's hand-written `Debug` redacts the key for the
|
||||
//! one place a value could still reach a line — an error context chain.
|
||||
//!
|
||||
//! The consumer sits inside the container: `nix/agent-modules/bao.nix`'s
|
||||
//! `hive-agent-bao-identity.service` logs in with what lands here and fails
|
||||
//! loudly when it cannot.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use hive_priv_sock::CredentialMount;
|
||||
use swarm_secret_client::{SecretStore, client, mtls};
|
||||
|
||||
/// systemd credential ids the agent's identity arrives under inside the
|
||||
/// container. `nix/agent-modules/bao.nix` spells the same three in its
|
||||
/// `LoadCredential=`; neither side can discover the other's, so a rename here
|
||||
/// is a rename there.
|
||||
///
|
||||
/// No `.pem` suffix: `hive-priv` refuses a credential name containing a dot,
|
||||
/// since the name is interpolated into `--load-credential=<name>:<path>`.
|
||||
const CERT_CREDENTIAL: &str = "hive-agent-bao-cert";
|
||||
const KEY_CREDENTIAL: &str = "hive-agent-bao-key";
|
||||
/// The authority the **store's own listener** is verified against — not the
|
||||
/// authority that issued the agent's leaf, which no reader in the container
|
||||
/// needs and which is therefore not delivered at all.
|
||||
///
|
||||
/// Copied into the staging dir rather than forwarded from the hive's own
|
||||
/// `BAO_CACERT`: that path is this daemon's per-unit credential directory,
|
||||
/// which systemd removes when `hive-c0re` stops. An nspawn config pointing at
|
||||
/// it would leave a container unable to start whenever the daemon was down.
|
||||
const SERVER_CA_CREDENTIAL: &str = "hive-agent-bao-server-ca";
|
||||
|
||||
/// Collect `agent`'s store identity and return the credentials to forward.
|
||||
///
|
||||
/// Empty whenever this hive cannot answer the question — no store configured,
|
||||
/// no hive name to log in as, nothing published for this agent yet. That is
|
||||
/// the ordinary state of an agent created before the swarm minted identities,
|
||||
/// and an empty list keeps its container starting exactly as it does today.
|
||||
/// The loud failure belongs inside the container, where the option that asked
|
||||
/// for an identity is set; refusing to write an nspawn config here would take
|
||||
/// down agents that never wanted one.
|
||||
///
|
||||
/// Already-staged files survive a failed collection. A store that is briefly
|
||||
/// unreachable then delivers the identity from the last successful read rather
|
||||
/// than none at all — the same call as the queue reader's, which leaves its
|
||||
/// files alone when the store says nothing.
|
||||
pub async fn stage(agent: &str) -> Vec<CredentialMount> {
|
||||
let dir = crate::paths::agent_identity_dir(agent);
|
||||
match collect(agent, &dir).await {
|
||||
Ok(()) => {}
|
||||
Err(Skipped::NotConfigured(why)) => {
|
||||
tracing::info!(%agent, "no swarm store identity for this agent: {why}");
|
||||
}
|
||||
Err(Skipped::Failed(e)) => {
|
||||
// `warn`, not `error`: whatever is already staged still goes in,
|
||||
// and the container's own check is what decides whether that is
|
||||
// good enough.
|
||||
tracing::warn!(
|
||||
%agent, error = ?e,
|
||||
"collecting this agent's store identity failed; delivering whatever was staged before"
|
||||
);
|
||||
}
|
||||
}
|
||||
mounts(&dir)
|
||||
}
|
||||
|
||||
/// Why a collection produced nothing, split by whether an operator has
|
||||
/// anything to fix: a hive with no store wired up says so once at `info`,
|
||||
/// a hive that has one and could not read says so at `warn`.
|
||||
enum Skipped {
|
||||
NotConfigured(&'static str),
|
||||
Failed(anyhow::Error),
|
||||
}
|
||||
|
||||
/// Read the published identity and write it into `dir`.
|
||||
async fn collect(agent: &str, dir: &Path) -> Result<(), Skipped> {
|
||||
// The hive's name is the cert-auth role it logs in as: `glue-bao-tls.nix`
|
||||
// mints this host's client certificate with the hive name as its CN and a
|
||||
// bao cert role matches on CN, so the two share a name by construction —
|
||||
// the same reasoning `swarm_status::handle_credential_notice` records for
|
||||
// the other read this hive does on an agent's behalf.
|
||||
let Some(hive) = crate::container_view::hive_swarm_names().0 else {
|
||||
return Err(Skipped::NotConfigured(
|
||||
"HYPERHIVE_HIVE_NAME is unset, so this hive has no role to log in as",
|
||||
));
|
||||
};
|
||||
// Checked before connecting so a host with no store at all costs a lookup
|
||||
// rather than a timeout, and reports the cause instead of the symptom.
|
||||
if std::env::var_os(client::ENV_ADDR).is_none_or(|v| v.is_empty()) {
|
||||
return Err(Skipped::NotConfigured(
|
||||
"this hive has no swarm secret store configured",
|
||||
));
|
||||
}
|
||||
|
||||
let path = mtls::identity_path(agent).map_err(|e| Skipped::Failed(e.into()))?;
|
||||
let store = SecretStore::from_env(&hive)
|
||||
.await
|
||||
.map_err(|e| Skipped::Failed(anyhow::Error::new(e).context("connecting to the store")))?;
|
||||
let credential: mtls::Credential = store.read(&path).await.map_err(|e| {
|
||||
Skipped::Failed(anyhow::Error::new(e).context(format!("reading {path} from the store")))
|
||||
})?;
|
||||
|
||||
write_identity(dir, &credential).map_err(Skipped::Failed)
|
||||
}
|
||||
|
||||
/// Write the two private files, replacing whatever was there.
|
||||
///
|
||||
/// `0600` from the moment each file exists rather than after a `chmod`: the
|
||||
/// key is written through a handle opened with that mode, so there is no
|
||||
/// instant where it sits on disk world-readable.
|
||||
fn write_identity(dir: &Path, credential: &mtls::Credential) -> anyhow::Result<()> {
|
||||
use anyhow::Context as _;
|
||||
use std::io::Write as _;
|
||||
use std::os::unix::fs::{OpenOptionsExt as _, PermissionsExt as _};
|
||||
|
||||
std::fs::create_dir_all(dir).with_context(|| format!("create {}", dir.display()))?;
|
||||
// The directory too: its own mode is what stops anything but `hive-core`
|
||||
// and root from reaching a file inside it by name.
|
||||
std::fs::set_permissions(dir, std::fs::Permissions::from_mode(0o700))
|
||||
.with_context(|| format!("chmod 0700 {}", dir.display()))?;
|
||||
|
||||
for (name, bytes) in [
|
||||
(CERT_CREDENTIAL, credential.cert.as_bytes()),
|
||||
(KEY_CREDENTIAL, credential.key.as_bytes()),
|
||||
] {
|
||||
let file = dir.join(name);
|
||||
let mut handle = std::fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.create(true)
|
||||
.truncate(true)
|
||||
.mode(0o600)
|
||||
.open(&file)
|
||||
.with_context(|| format!("open {}", file.display()))?;
|
||||
// The error carries the path and never the bytes — the one line in
|
||||
// this function where a value could otherwise be interpolated.
|
||||
handle
|
||||
.write_all(bytes)
|
||||
.with_context(|| format!("write {}", file.display()))?;
|
||||
}
|
||||
|
||||
// Public material, and `0644` says so — it is the same bundle every host
|
||||
// unit in the tree already points `BAO_CACERT` at. Absent means the
|
||||
// container falls back to its own trust store, which is what a deployment
|
||||
// with a real CA wants.
|
||||
let server_ca = std::env::var_os(client::ENV_CACERT)
|
||||
.map(PathBuf::from)
|
||||
.filter(|p| p.is_file());
|
||||
if let Some(source) = server_ca {
|
||||
let staged = dir.join(SERVER_CA_CREDENTIAL);
|
||||
std::fs::copy(&source, &staged)
|
||||
.with_context(|| format!("copy {} to {}", source.display(), staged.display()))?;
|
||||
std::fs::set_permissions(&staged, std::fs::Permissions::from_mode(0o644))
|
||||
.with_context(|| format!("chmod 0644 {}", staged.display()))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The credentials to forward, given what is actually staged.
|
||||
///
|
||||
/// Both private files or neither: a container that received a certificate
|
||||
/// without its key would start a login check that cannot pass, and the
|
||||
/// resulting failure would name the wrong problem.
|
||||
fn mounts(dir: &Path) -> Vec<CredentialMount> {
|
||||
let cert = dir.join(CERT_CREDENTIAL);
|
||||
let key = dir.join(KEY_CREDENTIAL);
|
||||
if !cert.is_file() || !key.is_file() {
|
||||
return Vec::new();
|
||||
}
|
||||
let mut out = vec![
|
||||
CredentialMount {
|
||||
name: CERT_CREDENTIAL.to_owned(),
|
||||
host_path: cert.to_string_lossy().into_owned(),
|
||||
},
|
||||
CredentialMount {
|
||||
name: KEY_CREDENTIAL.to_owned(),
|
||||
host_path: key.to_string_lossy().into_owned(),
|
||||
},
|
||||
];
|
||||
// Absent means the container verifies the store's listener against its own
|
||||
// trust store, which is what a deployment with a real CA wants — the same
|
||||
// optional shape every `BAO_CACERT` site in the nix tree already has.
|
||||
let ca = dir.join(SERVER_CA_CREDENTIAL);
|
||||
if ca.is_file() {
|
||||
out.push(CredentialMount {
|
||||
name: SERVER_CA_CREDENTIAL.to_owned(),
|
||||
host_path: ca.to_string_lossy().into_owned(),
|
||||
});
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{CERT_CREDENTIAL, KEY_CREDENTIAL, SERVER_CA_CREDENTIAL, mounts, write_identity};
|
||||
use swarm_secret_client::mtls;
|
||||
|
||||
fn credential() -> mtls::Credential {
|
||||
mtls::Credential {
|
||||
cert: "-----BEGIN CERTIFICATE-----\nleaf\n".to_owned(),
|
||||
key: "-----BEGIN PRIVATE KEY-----\nSUPER-SECRET\n".to_owned(),
|
||||
ca: "-----BEGIN CERTIFICATE-----\nauthority\n".to_owned(),
|
||||
}
|
||||
}
|
||||
|
||||
/// The property the whole delivery rests on: the key lands unreadable to
|
||||
/// anything but its owner, so the only way into the container is the
|
||||
/// credential mechanism.
|
||||
#[test]
|
||||
fn the_staged_key_is_private_the_moment_it_exists() {
|
||||
use std::os::unix::fs::PermissionsExt as _;
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let staged = dir.path().join("iris");
|
||||
write_identity(&staged, &credential()).expect("writes");
|
||||
for name in [CERT_CREDENTIAL, KEY_CREDENTIAL] {
|
||||
let mode = std::fs::metadata(staged.join(name))
|
||||
.expect("staged")
|
||||
.permissions()
|
||||
.mode();
|
||||
assert_eq!(mode & 0o777, 0o600, "{name} must be 0600, got {mode:o}");
|
||||
}
|
||||
let dir_mode = std::fs::metadata(&staged)
|
||||
.expect("staged dir")
|
||||
.permissions()
|
||||
.mode();
|
||||
assert_eq!(dir_mode & 0o777, 0o700, "got {dir_mode:o}");
|
||||
}
|
||||
|
||||
/// Re-staging is what happens on every rebuild, so a second write must
|
||||
/// leave the file holding only the new value — a shorter certificate
|
||||
/// written over a longer one without truncation is a file that parses as
|
||||
/// neither.
|
||||
#[test]
|
||||
fn re_staging_replaces_rather_than_overwrites_in_place() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let staged = dir.path().join("iris");
|
||||
let mut long = credential();
|
||||
long.cert = "-".repeat(400);
|
||||
write_identity(&staged, &long).expect("writes");
|
||||
write_identity(&staged, &credential()).expect("re-writes");
|
||||
let on_disk = std::fs::read_to_string(staged.join(CERT_CREDENTIAL)).expect("read back");
|
||||
assert_eq!(on_disk, credential().cert);
|
||||
}
|
||||
|
||||
/// Half a delivery is worse than none: the container's check would fail
|
||||
/// on the missing key and report a login problem instead of a delivery
|
||||
/// one. Same both-or-neither rule the queue credential states.
|
||||
#[test]
|
||||
fn a_certificate_without_its_key_forwards_nothing() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
assert!(mounts(dir.path()).is_empty());
|
||||
std::fs::write(dir.path().join(CERT_CREDENTIAL), "leaf").expect("write cert");
|
||||
assert!(mounts(dir.path()).is_empty());
|
||||
}
|
||||
|
||||
/// The store's own authority rides along when there is one, so a
|
||||
/// container verifying a self-signed listener has something to verify
|
||||
/// against — and the absence of it is not a reason to withhold the
|
||||
/// identity.
|
||||
#[test]
|
||||
fn a_staged_server_authority_is_forwarded_beside_the_identity() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
std::fs::write(dir.path().join(CERT_CREDENTIAL), "leaf").expect("write cert");
|
||||
std::fs::write(dir.path().join(KEY_CREDENTIAL), "private").expect("write key");
|
||||
let without: Vec<String> = mounts(dir.path()).into_iter().map(|c| c.name).collect();
|
||||
assert_eq!(without, [CERT_CREDENTIAL, KEY_CREDENTIAL]);
|
||||
|
||||
std::fs::write(dir.path().join(SERVER_CA_CREDENTIAL), "authority").expect("write ca");
|
||||
let with: Vec<String> = mounts(dir.path()).into_iter().map(|c| c.name).collect();
|
||||
assert_eq!(
|
||||
with,
|
||||
[CERT_CREDENTIAL, KEY_CREDENTIAL, SERVER_CA_CREDENTIAL]
|
||||
);
|
||||
}
|
||||
|
||||
/// The names are the agreement with `nix/agent-modules/bao.nix`, and
|
||||
/// `hive-priv` refuses any of them that could close the flag it is
|
||||
/// interpolated into.
|
||||
#[test]
|
||||
fn every_credential_id_is_one_hive_priv_will_accept() {
|
||||
for name in [CERT_CREDENTIAL, KEY_CREDENTIAL, SERVER_CA_CREDENTIAL] {
|
||||
assert!(
|
||||
name.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'),
|
||||
"{name} must be [A-Za-z0-9_-] only"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -253,7 +253,14 @@ async fn set_nspawn_flags(
|
|||
// here, and forwarded *as a credential* rather than a bind for the
|
||||
// reason `queue_agent_credentials` records.
|
||||
let queue_credential_dir = std::env::var_os(QUEUE_CREDENTIAL_DIR_ENV).map(PathBuf::from);
|
||||
let load_creds = queue_agent_credentials(agent_name, queue_credential_dir.as_deref());
|
||||
let mut load_creds = queue_agent_credentials(agent_name, queue_credential_dir.as_deref());
|
||||
|
||||
// The agent's own identity *at* the swarm secret store, collected under
|
||||
// this hive's certificate and forwarded the same way and for the same
|
||||
// reason. Staged here rather than at spawn because this is the one place
|
||||
// that writes the container's credential list — see
|
||||
// `super::agent_identity` for the whole hop.
|
||||
load_creds.extend(super::agent_identity::stage(agent_name).await);
|
||||
|
||||
let mut binds: Vec<BindMount> = vec![
|
||||
BindMount {
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
//! `nixos-container` lifecycle + per-agent config flake generation.
|
||||
|
||||
mod agent_identity;
|
||||
mod git;
|
||||
mod host_config;
|
||||
mod setup;
|
||||
|
|
|
|||
|
|
@ -706,6 +706,11 @@ const FORWARDED_VARS: &[&str] = &[
|
|||
// address, which inside a container is the agent itself.
|
||||
"HIVE_AGENT_NATS_URL",
|
||||
"HIVE_AGENT_OIDC_TOKEN_ENDPOINT",
|
||||
// Where the swarm secret store listens. The hive's own `BAO_ADDR` value,
|
||||
// forwarded under a separate name so the container never inherits the
|
||||
// hive's store environment wholesale — the address is the only part of it
|
||||
// an agent may have, the certificate beside it being the hive's own.
|
||||
"HIVE_AGENT_BAO_ADDR",
|
||||
// Where an agent's `swarm-logs` queries, computed by the host for the
|
||||
// same reason as the two above: it is a gateway address, and a container
|
||||
// cannot derive it from anything it holds.
|
||||
|
|
@ -747,6 +752,12 @@ const FORWARDED_VAR_OPTIONS: &[(&str, &str)] = &[
|
|||
"HIVE_AGENT_OIDC_TOKEN_ENDPOINT",
|
||||
"services.hyperhive.agent.queue.tokenEndpoint",
|
||||
),
|
||||
// Read at build time to decide whether the in-container identity check is
|
||||
// generated at all — see `nix/agent-modules/bao.nix`. Both halves wired
|
||||
// for this map's own reason: the unit bakes the address into its own
|
||||
// environment, so an agent whose option says no store and whose env says
|
||||
// otherwise would have a check that is simply not there.
|
||||
("HIVE_AGENT_BAO_ADDR", "services.hyperhive.agent.bao.addr"),
|
||||
// Read at build time to decide whether `swarm-logs` is installed at all —
|
||||
// see `nix/agent-modules/logs.nix`. Both halves wired for this map's own
|
||||
// reason: an agent whose option says no log store and whose env says
|
||||
|
|
|
|||
|
|
@ -102,6 +102,32 @@ pub fn forge_repo_creation_disabled_marker(name: &str) -> PathBuf {
|
|||
forge_dir().join(format!("repo-creation-disabled-{name}"))
|
||||
}
|
||||
|
||||
/// `agent-identity/` — one subdir per agent holding the store identity this
|
||||
/// hive collected for it. Staging only: the files exist so `systemd-nspawn`
|
||||
/// has something to `--load-credential` from, and nothing on the host ever
|
||||
/// reads them back.
|
||||
///
|
||||
/// Under the state root rather than `/run` deliberately. The forward happens
|
||||
/// when a container's nspawn config is (re)written, and a container can be
|
||||
/// restarted long after that; material that vanished with `/run` would leave
|
||||
/// a boot where the agent's identity is simply absent and the only symptom is
|
||||
/// a unit inside the container refusing to start.
|
||||
#[must_use]
|
||||
pub fn agent_identity_root() -> PathBuf {
|
||||
state_root().join("agent-identity")
|
||||
}
|
||||
|
||||
/// `agent-identity/<name>` — one agent's staged store identity.
|
||||
///
|
||||
/// ⚠️ Deliberately **not** under `agents/<name>`: that tree is bind-mounted
|
||||
/// into the container (and a parent's, for a child), so a private key placed
|
||||
/// there would be readable by the agent as a plain file, bypassing the
|
||||
/// credential mechanism that exists to control exactly that.
|
||||
#[must_use]
|
||||
pub fn agent_identity_dir(name: &str) -> PathBuf {
|
||||
agent_identity_root().join(name)
|
||||
}
|
||||
|
||||
/// `matrix/` — host-side matrix provisioning state (admin token, hive
|
||||
/// Space room id, per-agent password creds). The shared registration
|
||||
/// token is bind-mounted into the tuwunel container via nix and stays
|
||||
|
|
|
|||
Loading…
Reference in a new issue