mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re pulls closures through the nix daemon; hive-priv verifies the swarm signature before applying

Browse source
This commit is contained in:
atlas 2026-11-07 13:20:55 +01:00 • committed by mara
commit ecd4445680
3 changed files with 1257 additions and 584 deletions
Download patch file Download diff file Expand all files Collapse all files

719
hive-priv/src/closure.rs
View file

@ -1,42 +1,34 @@
//! `RealiseClosure`: download the closure the swarm built for an agent from
//! the swarm's binary cache into the store, without ever building.
//! The check on a closure before it is applied, and the two small verbs around
//! the swarm cache.
//!
//! What this accepts is decided by root-owned sources alone: the cache's
//! address is the host's configuration, handed to the unit in
//! [`CACHE_URL_ENV`], and the cache's public key and the pull token's netrc are
//! files a root unit keeps in [`ROOT_FACTS_DIR`]. Nothing in the request, and
//! nothing hive-c0re (unprivileged) writes, chooses a cache or a key. They go to
//! nix's daemon as per-call settings, which the daemon applies for a trusted
//! client such as this one (root), so the host's nix configuration is untouched.
//! hive-c0re (unprivileged) downloads the swarm's closure through the host's
//! nix daemon. `CreateContainer` and `UpdateContainer` then apply a toplevel
//! only when it is in the store and carries a signature by the swarm cache's
//! key, which `nix store verify` checks against the key in [`ROOT_FACTS_DIR`].
//! The key is a file a root unit keeps; nothing in the request, and nothing
//! hive-c0re writes, chooses it.
use std::os::unix::fs::MetadataExt as _;
use std::path::{Path, PathBuf};
use std::path::Path;
use anyhow::{Context as _, Result, bail};
use hive_priv_sock::{SIGNATURE_REFUSED, valid_closure_path};
use hive_priv_sock::valid_closure_path;
use crate::container::{BoundedRun, run_bounded};
use crate::shared::validate_agent_name;
/// The unit's environment variable naming the cache: `https://<domain>/<cache>`
/// from the host's `services.hyperhive.swarm.attic` options.
pub(crate) const CACHE_URL_ENV: &str = "HIVE_PRIV_CACHE_URL";
use crate::shared::{RealSpawn, Spawn, validate_agent_name};
/// Where the root unit `hive-priv-cache-facts` keeps `public-key` and `netrc`.
/// Not `/run/hyperhive/attic`, which hive-c0re writes.
const ROOT_FACTS_DIR: &str = "/run/hyperhive-priv/attic";
/// Wall-clock bound on one download. A stalled transfer is bounded by nix's
/// own `stalled-download-timeout`; this covers the rest.
const REALISE_TIMEOUT: std::time::Duration = std::time::Duration::from_mins(30);
/// The root unit that reads this hive's cache facts from the store.
const FACTS_UNIT: &str = "hive-priv-cache-facts.service";
/// The cache this host trusts.
#[derive(Debug, PartialEq, Eq)]
struct Cache {
substituter: String,
public_key: String,
netrc: PathBuf,
}
/// Wall-clock bound on one `nix store verify`, which reads a path's metadata.
const VERIFY_TIMEOUT: std::time::Duration = std::time::Duration::from_mins(1);
/// Wall-clock bound on the facts unit: a store login and three small writes.
const REFRESH_TIMEOUT: std::time::Duration = std::time::Duration::from_mins(1);
/// Whether `path` is a regular file or directory owned by `owner` that nobody
/// else can read or write.
@ -64,305 +56,180 @@ fn private_to(path: &Path, owner: u32, directory: bool) -> Result<()> {
Ok(())
}
impl Cache {
/// The cache at `url`, trusting the key in `dir`.
///
/// # Errors
/// No `url`, one that is not a plain `https` address (the token would go
/// out in clear), a `dir` or file that is missing, not owned by `owner` or
/// open to others, or a key that is not `<name>:<key>`.
fn read(url: Option<&str>, dir: &Path, owner: u32) -> Result<Self> {
let url = url
.map(str::trim)
.filter(|u| !u.is_empty())
.context("this host names no cache")?;
if !url.starts_with("https://") || url.contains(char::is_whitespace) {
bail!("the cache address is not an https URL");
}
private_to(dir, owner, true)?;
let key_file = dir.join("public-key");
private_to(&key_file, owner, false)?;
let netrc = dir.join("netrc");
private_to(&netrc, owner, false)?;
let key = std::fs::read_to_string(&key_file)
.with_context(|| format!("reading {}", key_file.display()))?;
let key = key.trim();
if key.contains(char::is_whitespace)
|| key
.split_once(':')
.is_none_or(|(name, key)| name.is_empty() || key.is_empty())
{
bail!("the cache's public key is not <name>:<key>");
}
Ok(Self {
substituter: url.to_owned(),
public_key: key.to_owned(),
netrc,
})
}
}
/// The `nix-store` arguments that fetch `store_path` and nothing else.
///
/// - `--max-jobs 0` forbids a local build, so a path the cache lacks fails.
/// - `substituters` and `trusted-public-keys` are replaced, not extended: the
/// swarm cache is the only source and its key the only one that counts.
/// - `require-sigs true` makes nix refuse a path no trusted key signed.
fn realise_args(store_path: &str, cache: &Cache) -> Vec<String> {
let mut args: Vec<String> = ["--realise", store_path, "--max-jobs", "0"]
.map(str::to_owned)
.into();
let netrc = cache.netrc.display().to_string();
for (name, value) in [
("substituters", cache.substituter.as_str()),
("trusted-public-keys", cache.public_key.as_str()),
("require-sigs", "true"),
("netrc-file", netrc.as_str()),
] {
args.extend(["--option".to_owned(), name.to_owned(), value.to_owned()]);
}
args
}
/// Why nix did not realise the path, from its last stderr line. A refusal for
/// want of a signature is named, so the caller can tell it from a cache that
/// is down.
fn reason(stderr: &str) -> String {
let last = stderr.lines().last().unwrap_or("").trim();
if last.to_lowercase().contains("signature") {
format!("{SIGNATURE_REFUSED}: {last}")
} else {
last.to_owned()
}
}
/// `RealiseClosure`.
/// The swarm cache's public key in `dir`, as `<name>:<key>`.
///
/// # Errors
/// Bad arguments, or `not available: <reason>` for anything that left the path
/// out of the store: no cache or key on this host, nix failing or timing out.
pub(crate) async fn realise_closure(agent: &str, store_path: &str) -> Result<(String, String)> {
let url = std::env::var(CACHE_URL_ENV).ok();
let done = realise_with(
url.as_deref(),
Path::new(ROOT_FACTS_DIR),
0,
agent,
store_path,
)
.await?;
if let Ok(mut realised) = REALISED.lock() {
remember(&mut realised, agent, store_path);
/// A `dir` or key file that is missing, not owned by `owner` or open to
/// others, or a key that is not `<name>:<key>`.
fn swarm_key(dir: &Path, owner: u32) -> Result<String> {
private_to(dir, owner, true)?;
let key_file = dir.join("public-key");
private_to(&key_file, owner, false)?;
let key = std::fs::read_to_string(&key_file)
.with_context(|| format!("reading {}", key_file.display()))?;
let key = key.trim();
if key.contains(char::is_whitespace)
|| key
.split_once(':')
.is_none_or(|(name, key)| name.is_empty() || key.is_empty())
{
bail!("the cache's public key is not <name>:<key>");
}
Ok(done)
Ok(key.to_owned())
}
/// The store path each agent's last successful `RealiseClosure` put in the
/// store. `CreateContainer` / `UpdateContainer` apply only that path for that
/// agent. hive-priv is a long-running process, so the entry outlives the
/// calls between the download and the apply; a restart in between empties it
/// and the apply is refused.
static REALISED: std::sync::Mutex<Vec<(String, String)>> = std::sync::Mutex::new(Vec::new());
fn remember(realised: &mut Vec<(String, String)>, agent: &str, store_path: &str) {
realised.retain(|(a, _)| a != agent);
realised.push((agent.to_owned(), store_path.to_owned()));
}
/// The path recorded for `agent`.
fn lookup<'a>(realised: &'a [(String, String)], agent: &str) -> Option<&'a str> {
realised
.iter()
.find(|(a, _)| a == agent)
.map(|(_, p)| p.as_str())
}
/// `toplevel` when it is a store path and the one `agent`'s last
/// `RealiseClosure` returned (`realised`); else the refusal, which is also
/// logged at error level with the agent and both paths.
/// The `nix` arguments that check `toplevel` is in the store and signed by
/// `key`, and by no other key.
///
/// The request comes from hive-c0re, which is not root. A signed path is not
/// enough: it could be another agent's closure or the host's own.
///
/// # Errors
/// `toplevel` is not a store path, or is not `realised`.
pub(crate) fn applicable<'a>(
agent: &str,
toplevel: &'a str,
realised: Option<&str>,
) -> Result<&'a str> {
/// - `--no-contents` checks signatures and validity, not the NAR hash.
/// - `--sigs-needed 1` with `trusted-public-keys` replaced by `key` means one
/// signature by the swarm cache's key; the host's other keys do not count.
fn verify_args(toplevel: &str, key: &str) -> Vec<String> {
[
"--extra-experimental-features",
"nix-command",
"store",
"verify",
"--no-contents",
"--sigs-needed",
"1",
"--option",
"trusted-public-keys",
key,
toplevel,
]
.map(str::to_owned)
.into()
}
/// What `nix store verify` reports through its exit status, as a bit set: 2 is
/// a path no trusted key signed, 4 a path that is not in the store.
fn verdict(code: Option<i32>, toplevel: &str, stderr: &str) -> Result<()> {
match code {
Some(0) => Ok(()),
Some(c) if c & 4 != 0 => bail!("{toplevel} is not in the store"),
Some(c) if c & 2 != 0 => bail!("{toplevel} has no signature by the swarm cache's key"),
_ => bail!(
"nix store verify failed: {}",
stderr.lines().last().unwrap_or("").trim()
),
}
}
/// Whether `toplevel` is in the store with a signature by the key in `dir`.
async fn check_signed(spawn: &impl Spawn, dir: &Path, owner: u32, toplevel: &str) -> Result<()> {
if !valid_closure_path(toplevel) {
bail!("{toplevel:?} is not a store path");
}
if realised == Some(toplevel) {
return Ok(toplevel);
let key = swarm_key(dir, owner)?;
let mut cmd = spawn.command("nix");
cmd.args(verify_args(toplevel, &key));
match run_bounded(cmd, VERIFY_TIMEOUT)
.await
.context("running nix store verify")?
{
BoundedRun::Exited { status, stderr } => verdict(status.code(), toplevel, &stderr),
BoundedRun::TimedOut => bail!(
"nix store verify timed out after {}s; killed it",
VERIFY_TIMEOUT.as_secs()
),
}
let realised = realised.unwrap_or("none");
tracing::error!(
%agent, requested = %toplevel, %realised,
"refusing to apply a closure that is not the one realised for the agent"
);
bail!("{agent}: {toplevel} is not the closure realised for this agent ({realised})")
}
/// [`applicable`] against what `RealiseClosure` recorded for `agent`.
/// `toplevel` when it is a store path that is in the store with a signature by
/// the swarm cache's key; else the refusal, which is also logged at error
/// level with the agent and the path.
///
/// The request comes from hive-c0re, which is not root and can ask for any
/// path of that shape. A path in the store is not enough: a build of the host's
/// own, or a path from another cache, carries no signature by this key.
///
/// # Errors
/// As [`applicable`].
pub(crate) fn applicable_for<'a>(agent: &str, toplevel: &'a str) -> Result<&'a str> {
let realised = REALISED
.lock()
.ok()
.and_then(|r| lookup(&r, agent).map(str::to_owned));
applicable(agent, toplevel, realised.as_deref())
/// `toplevel` is not a store path, is not in the store, or carries no
/// signature by the swarm cache's key, or the key cannot be read.
pub(crate) async fn applicable_for<'a>(agent: &str, toplevel: &'a str) -> Result<&'a str> {
applicable_with(&RealSpawn, Path::new(ROOT_FACTS_DIR), 0, agent, toplevel).await
}
/// [`realise_closure`] with the cache's address and the key's directory and
/// owner named, so a test can stand in for the host.
async fn realise_with(
url: Option<&str>,
async fn applicable_with<'a>(
spawn: &impl Spawn,
dir: &Path,
owner: u32,
agent: &str,
store_path: &str,
) -> Result<(String, String)> {
toplevel: &'a str,
) -> Result<&'a str> {
match check_signed(spawn, dir, owner, toplevel).await {
Ok(()) => Ok(toplevel),
Err(e) => {
tracing::error!(
%agent, requested = %toplevel, error = %format!("{e:#}"),
"refusing to apply a closure the swarm cache did not sign"
);
bail!("{agent}: refusing {toplevel}: {e:#}")
}
}
}
/// `RealiseClosure`, from an older hive-c0re: nothing is downloaded. `Ok` when
/// the path is already what an apply requires, else `not available: <reason>`.
///
/// # Errors
/// Bad arguments, or `not available: <reason>`.
pub(crate) async fn realise_closure(agent: &str, store_path: &str) -> Result<(String, String)> {
validate_agent_name(agent)?;
if !valid_closure_path(store_path) {
bail!("{store_path:?} is not a store path");
}
let outcome = async {
let cache = Cache::read(url, dir, owner)?;
let mut cmd = tokio::process::Command::new("nix-store");
cmd.args(realise_args(store_path, &cache));
match run_bounded(cmd, REALISE_TIMEOUT)
.await
.context("running nix-store")?
{
BoundedRun::Exited { status, .. } if status.success() => Ok(()),
BoundedRun::Exited { stderr, .. } => bail!("{}", reason(&stderr)),
BoundedRun::TimedOut => bail!(
"nix-store timed out after {}s; killed it",
REALISE_TIMEOUT.as_secs()
),
}
match check_signed(&RealSpawn, Path::new(ROOT_FACTS_DIR), 0, store_path).await {
Ok(()) => Ok(("present".to_owned(), String::new())),
Err(e) => bail!(
"not available: {e:#}; hive-priv no longer downloads, hive-c0re pulls through the nix daemon"
),
}
.await;
match outcome {
Ok(()) => {
tracing::info!(%agent, %store_path, "substituted the agent's closure");
Ok(("substituted".to_owned(), String::new()))
}
Err(e) => {
let reason = format!("{e:#}");
if reason.contains(SIGNATURE_REFUSED) {
tracing::error!(%agent, %store_path, %reason, "the cache's closure was refused");
}
bail!("not available: {reason}")
}
/// `RefreshCacheFacts`: start the root unit that reads this hive's cache facts
/// from the store and writes the nix.conf fragment, and wait for it. The unit
/// is fixed here, not named by the request.
///
/// # Errors
/// The unit failed to start or did not finish within [`REFRESH_TIMEOUT`].
pub(crate) async fn refresh_cache_facts() -> Result<(String, String)> {
refresh_with(&RealSpawn, REFRESH_TIMEOUT).await
}
async fn refresh_with(spawn: &impl Spawn, limit: std::time::Duration) -> Result<(String, String)> {
let mut cmd = spawn.command("systemctl");
cmd.args(["start", FACTS_UNIT]);
match run_bounded(cmd, limit)
.await
.with_context(|| format!("running systemctl start {FACTS_UNIT}"))?
{
BoundedRun::Exited { status, .. } if status.success() => {
tracing::info!("refreshed the cache facts");
Ok((String::new(), String::new()))
}
BoundedRun::Exited { status, stderr } => bail!(
"systemctl start {FACTS_UNIT} exited {status}: {}",
stderr.lines().last().unwrap_or("").trim()
),
BoundedRun::TimedOut => bail!(
"{FACTS_UNIT} did not finish within {}s; killed the wait",
limit.as_secs()
),
}
}
#[cfg(test)]
mod tests {
use std::os::unix::fs::PermissionsExt as _;
use std::path::PathBuf;
use super::*;
use crate::shared::test_support::scratch;
use crate::shared::test_support::{ScriptSpawn, scratch};
const PATH: &str = "/nix/store/0123456789abcdfghijklmnpqrsvwxyz-nixos-system-atlas";
const URL: &str = "https://cache.example/swarm";
fn cache() -> Cache {
Cache {
substituter: URL.to_owned(),
public_key: "swarm:AAAA".to_owned(),
netrc: PathBuf::from("/run/hyperhive-priv/attic/netrc"),
}
}
const OTHER: &str = "/nix/store/zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz-nixos-system-iris";
#[test]
fn the_path_realised_for_the_agent_is_applied() {
assert_eq!(applicable("atlas", PATH, Some(PATH)).unwrap(), PATH);
}
/// Another agent's realised path, a valid-shaped path never realised, and
/// a restart that emptied the record are all refused, naming the agent
/// and both paths.
#[test]
fn any_other_path_is_refused() {
for (case, realised) in [
("another path was realised for the agent", Some(OTHER)),
("nothing was realised for the agent", None),
] {
let err = applicable("atlas", PATH, realised).unwrap_err();
let msg = format!("{err:#}");
assert!(
msg.contains("atlas")
&& msg.contains(PATH)
&& msg.contains(realised.unwrap_or("none")),
"{case}: {msg}"
);
}
}
#[test]
fn only_a_store_path_is_applied_even_when_it_was_recorded() {
for bad in [
"",
"main",
"/etc/passwd",
"/nix/store/x",
&format!("{PATH}/bin"),
] {
let err = applicable("atlas", bad, Some(bad)).unwrap_err();
assert!(format!("{err:#}").contains("not a store path"), "{bad}");
}
}
/// Each agent keeps its own entry, and a newer download replaces the
/// older one.
#[test]
fn a_record_is_per_agent_and_the_latest_wins() {
let mut realised = Vec::new();
remember(&mut realised, "atlas", PATH);
remember(&mut realised, "iris", OTHER);
assert_eq!(lookup(&realised, "atlas"), Some(PATH));
assert_eq!(lookup(&realised, "iris"), Some(OTHER));
remember(&mut realised, "atlas", OTHER);
assert_eq!(lookup(&realised, "atlas"), Some(OTHER));
assert_eq!(lookup(&realised, "nobody"), None);
assert_eq!(realised.len(), 2);
}
fn option<'a>(args: &'a [String], name: &str) -> Option<&'a str> {
args.windows(3)
.find(|w| w[0] == "--option" && w[1] == name)
.map(|w| w[2].as_str())
}
#[test]
fn the_command_never_builds() {
let args = realise_args(PATH, &cache());
assert_eq!(args[..2], ["--realise", PATH]);
let jobs = args.iter().position(|a| a == "--max-jobs").expect("flag");
assert_eq!(args[jobs + 1], "0");
}
#[test]
fn only_the_swarm_cache_and_its_key_count_and_signatures_are_required() {
let args = realise_args(PATH, &cache());
assert_eq!(option(&args, "substituters"), Some(URL));
assert_eq!(option(&args, "trusted-public-keys"), Some("swarm:AAAA"));
assert_eq!(option(&args, "require-sigs"), Some("true"));
assert_eq!(
option(&args, "netrc-file"),
Some("/run/hyperhive-priv/attic/netrc")
);
// The settings that would add to the defaults are not used.
assert!(!args.iter().any(|a| a.starts_with("extra-")), "{args:?}");
}
const KEY: &str = "swarm:AAAA";
/// The uid the test runs as, which stands in for root.
fn me() -> u32 {
@ -387,126 +254,169 @@ mod tests {
dir
}
#[test]
fn the_cache_is_the_hosts_address_and_the_root_files_key() {
let dir = facts("swarm:AAAA");
let cache = Cache::read(Some(URL), &dir, me()).expect("read");
assert_eq!(cache.substituter, URL);
assert_eq!(cache.public_key, "swarm:AAAA");
assert_eq!(cache.netrc, dir.join("netrc"));
/// A `nix` that behaves like `nix store verify --sigs-needed 1` over a store
/// holding `PATH`, signed by `KEY` only: exit 0 for that path under that key,
/// 2 for that path under any other key, 4 for any other path. It exits 99
/// for anything but the verify invocation, and logs its arguments.
fn nix(log: &Path) -> ScriptSpawn {
ScriptSpawn(format!(
r#"echo "$@" >>{log}
[ "$1 $2 $3 $4 $5" = "--extra-experimental-features nix-command store verify --no-contents" ] || exit 99
[ "$6 $7 $8 $9" = "--sigs-needed 1 --option trusted-public-keys" ] || exit 99
[ "${{11}}" = {PATH} ] || exit 4
[ "${{10}}" = {KEY} ] || exit 2
exit 0"#,
log = log.display()
))
}
/// A `substituter` or `public-key` file beside the facts, as hive-c0re
/// writes them, decides nothing: the address is the argument and the key is
/// the root file's.
#[test]
fn the_address_is_never_read_from_a_file_and_the_key_only_from_the_root_files() {
let dir = facts("swarm:ROOT");
put(
&dir.join("substituter"),
"https://evil.example/cache\n",
0o600,
#[tokio::test]
async fn a_toplevel_signed_by_the_swarm_key_is_applied() {
let log = scratch().join("calls");
let got = applicable_with(&nix(&log), &facts(KEY), me(), "atlas", PATH)
.await
.expect("signed");
assert_eq!(got, PATH);
// The key checked against is the one in the root file.
let calls = std::fs::read_to_string(&log).expect("log");
assert!(
calls.contains(&format!("trusted-public-keys {KEY} {PATH}")),
"{calls}"
);
let cache = Cache::read(Some(URL), &dir, me()).expect("read");
assert_eq!(cache.substituter, URL);
assert_eq!(cache.public_key, "swarm:ROOT");
// The directory hive-c0re writes is not the one this reads.
assert!(!ROOT_FACTS_DIR.starts_with("/run/hyperhive/"));
assert_eq!(ROOT_FACTS_DIR, "/run/hyperhive-priv/attic");
// And without an address from the host there is no cache, whatever the
// directory holds.
for none in [None, Some(""), Some(" ")] {
assert!(Cache::read(none, &dir, me()).is_err(), "{none:?}");
}
}
#[test]
fn a_file_that_is_not_the_roots_or_is_open_to_others_is_refused() {
let me = me();
// Control: the facts as the unit leaves them are accepted.
assert!(Cache::read(Some(URL), &facts("swarm:AAAA"), me).is_ok());
// Another owner than the one trusted.
assert!(Cache::read(Some(URL), &facts("swarm:AAAA"), me + 1).is_err());
// The key or the netrc readable by others, or the directory open.
for (name, mode) in [("public-key", 0o644), ("netrc", 0o644), ("netrc", 0o660)] {
let dir = facts("swarm:AAAA");
std::fs::set_permissions(dir.join(name), std::fs::Permissions::from_mode(mode))
.expect("chmod");
assert!(Cache::read(Some(URL), &dir, me).is_err(), "{name} {mode:o}");
}
let dir = facts("swarm:AAAA");
std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o755)).expect("chmod");
assert!(Cache::read(Some(URL), &dir, me).is_err());
// A symlink in place of the key.
let dir = facts("swarm:AAAA");
std::fs::remove_file(dir.join("public-key")).expect("rm");
std::os::unix::fs::symlink(dir.join("netrc"), dir.join("public-key")).expect("link");
assert!(Cache::read(Some(URL), &dir, me).is_err());
}
#[test]
fn missing_files_and_unusable_values_are_refused() {
let me = me();
for url in ["http://cache.example/swarm", "https://cache.example/a b"] {
/// A path no swarm key signed, one the store lacks, and a key file that
/// names another key are each refused, naming the agent and the path.
#[tokio::test]
async fn an_unsigned_or_absent_toplevel_is_refused() {
let log = scratch().join("calls");
let other = "/nix/store/zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz-nixos-system-iris";
for (case, dir, path, want) in [
("absent", facts(KEY), other, "is not in the store"),
(
"unsigned by the key on file",
facts("other:BBBB"),
PATH,
"has no signature by the swarm cache's key",
),
] {
let err = applicable_with(&nix(&log), &dir, me(), "atlas", path)
.await
.expect_err(case);
let msg = format!("{err:#}");
assert!(
Cache::read(Some(url), &facts("swarm:AAAA"), me).is_err(),
"{url}"
msg.contains("atlas") && msg.contains(path) && msg.contains(want),
"{case}: {msg}"
);
}
for key in ["nokeyname", "swarm: AAAA", ":AAAA", "swarm:"] {
assert!(Cache::read(Some(URL), &facts(key), me).is_err(), "{key:?}");
}
let dir = facts("swarm:AAAA");
std::fs::remove_file(dir.join("netrc")).expect("rm");
assert!(Cache::read(Some(URL), &dir, me).is_err());
assert!(Cache::read(Some(URL), &dir.join("nowhere"), me).is_err());
}
/// Missing facts reach the caller as `not available`, which a rebuild
/// reports as a closure that cannot be pulled.
/// A tool failure other than the two verdicts is a refusal too, with the
/// tool's last line.
#[tokio::test]
async fn without_the_root_files_the_answer_is_not_available() {
let nowhere = scratch().join("nowhere");
let err = realise_with(Some(URL), &nowhere, 0, "atlas", PATH)
async fn a_failing_verify_refuses() {
let spawn = ScriptSpawn("echo 'error: cannot connect' >&2; exit 1".to_owned());
let err = applicable_with(&spawn, &facts(KEY), me(), "atlas", PATH)
.await
.expect_err("no facts");
assert!(format!("{err:#}").starts_with("not available: "), "{err:#}");
let err = realise_with(None, &facts("swarm:AAAA"), me(), "atlas", PATH)
.await
.expect_err("no address");
assert!(format!("{err:#}").starts_with("not available: "), "{err:#}");
.expect_err("refused");
assert!(format!("{err:#}").contains("cannot connect"), "{err:#}");
}
/// The request carries an agent and a path and nothing that chooses a
/// cache: fields beyond those are dropped when it is read.
#[test]
fn the_request_cannot_carry_an_address_or_a_key() {
let line = r#"{"op":"realise_closure","agent":"atlas","store_path":"/nix/store/x",
"substituter":"https://evil.example/c","public_key":"evil:AAAA","netrc":"/tmp/x"}"#;
let hive_priv_sock::PrivRequest::RealiseClosure { agent, store_path } =
serde_json::from_str(line).expect("reads")
else {
panic!("not a RealiseClosure");
};
assert_eq!(
(agent.as_str(), store_path.as_str()),
("atlas", "/nix/store/x")
#[tokio::test]
async fn only_a_store_path_is_verified() {
// A stand-in that would accept anything, so only the shape check refuses.
let yes = ScriptSpawn("exit 0".to_owned());
for bad in [
"",
"main",
"/etc/passwd",
"/nix/store/x",
&format!("{PATH}/bin"),
] {
let err = applicable_with(&yes, &facts(KEY), me(), "atlas", bad)
.await
.expect_err(bad);
assert!(format!("{err:#}").contains("not a store path"), "{bad}");
}
// Control: the same stand-in accepts a store path.
assert!(
applicable_with(&yes, &facts(KEY), me(), "atlas", PATH)
.await
.is_ok()
);
}
#[test]
fn a_signature_refusal_is_named_and_another_failure_is_not() {
let refused =
reason("error: cannot add path 'x' because it lacks a signature by a trusted key");
assert!(refused.starts_with(SIGNATURE_REFUSED), "{refused}");
let down =
reason("warning: x\nerror: unable to download 'https://c/x.narinfo': Couldn't connect");
assert!(!down.contains(SIGNATURE_REFUSED), "{down}");
assert!(down.contains("Couldn't connect"), "{down}");
/// Without the root key there is nothing to verify against: refused, and
/// nix is not asked.
#[tokio::test]
async fn without_the_root_key_nothing_is_applied() {
let log = scratch().join("calls");
let nowhere = scratch().join("nowhere");
let err = applicable_with(&nix(&log), &nowhere, 0, "atlas", PATH)
.await
.expect_err("no key");
assert!(format!("{err:#}").contains("missing"), "{err:#}");
assert!(!log.exists(), "nix ran without a key");
}
#[test]
fn the_key_is_the_root_files_and_only_a_private_one() {
let me = me();
assert_eq!(swarm_key(&facts(KEY), me).unwrap(), KEY);
// Another owner than the one trusted.
assert!(swarm_key(&facts(KEY), me + 1).is_err());
// The key open to others, or the directory open.
let dir = facts(KEY);
std::fs::set_permissions(
dir.join("public-key"),
std::fs::Permissions::from_mode(0o644),
)
.expect("chmod");
assert!(swarm_key(&dir, me).is_err());
let dir = facts(KEY);
std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o755)).expect("chmod");
assert!(swarm_key(&dir, me).is_err());
// A symlink in place of the key.
let dir = facts(KEY);
std::fs::remove_file(dir.join("public-key")).expect("rm");
std::os::unix::fs::symlink(dir.join("netrc"), dir.join("public-key")).expect("link");
assert!(swarm_key(&dir, me).is_err());
for key in ["nokeyname", "swarm: AAAA", ":AAAA", "swarm:"] {
assert!(swarm_key(&facts(key), me).is_err(), "{key:?}");
}
// The directory hive-c0re writes is not the one this reads.
assert_eq!(ROOT_FACTS_DIR, "/run/hyperhive-priv/attic");
}
#[test]
fn the_command_counts_only_the_swarm_key() {
let args = verify_args(PATH, KEY);
assert_eq!(args.last().map(String::as_str), Some(PATH));
let at = args
.iter()
.position(|a| a == "--sigs-needed")
.expect("flag");
assert_eq!(args[at + 1], "1");
let at = args
.iter()
.position(|a| a == "trusted-public-keys")
.expect("option");
assert_eq!(args[at - 1], "--option");
assert_eq!(args[at + 1], KEY);
// The setting that would add to the host's keys is not used.
assert!(
!args.iter().any(|a| a.starts_with("extra-trusted")),
"{args:?}"
);
}
/// An older hive-c0re's download request downloads nothing: it succeeds
/// only for a path an apply would accept.
#[tokio::test]
async fn bad_arguments_are_refused_before_anything_runs() {
async fn the_older_download_request_answers_not_available_for_a_path_not_in_the_store() {
// The real `nix` is not on the path of the test, so this is refused for
// want of the root key before anything runs; the shape and name checks
// come first.
for (agent, path) in [
("Bad Agent", PATH),
("", PATH),
@ -517,5 +427,34 @@ mod tests {
let err = realise_closure(agent, path).await.expect_err("refused");
assert!(!format!("{err:#}").contains("not available"), "{err:#}");
}
let err = realise_closure("atlas", PATH).await.expect_err("no key");
let msg = format!("{err:#}");
assert!(
msg.starts_with("not available: ") && msg.contains("no longer downloads"),
"{msg}"
);
}
/// The refresh starts the fixed unit and nothing else, and says why it
/// failed or timed out.
#[tokio::test]
async fn the_refresh_starts_the_facts_unit_and_reports_a_failure() {
let log = scratch().join("calls");
let ok = ScriptSpawn(format!("echo \"$0 $@\" >>{}", log.display()));
refresh_with(&ok, REFRESH_TIMEOUT).await.expect("started");
assert_eq!(
std::fs::read_to_string(&log).expect("log").trim(),
"systemctl start hive-priv-cache-facts.service"
);
let failing = ScriptSpawn("echo 'Unit not found.' >&2; exit 5".to_owned());
let err = refresh_with(&failing, REFRESH_TIMEOUT)
.await
.expect_err("failed");
assert!(format!("{err:#}").contains("Unit not found."), "{err:#}");
let slow = ScriptSpawn("sleep 30".to_owned());
let err = refresh_with(&slow, std::time::Duration::from_millis(100))
.await
.expect_err("timed out");
assert!(format!("{err:#}").contains("did not finish"), "{err:#}");
}
}

24
hive-priv/src/container.rs
View file

@ -207,9 +207,10 @@ async fn destroy_container_with(host: &Host<impl Spawn>, name: &str) -> Result<(
}
/// Shared body for `CreateContainer` / `UpdateContainer`: validate the name
/// and that `toplevel` is the closure realised for the agent, and apply it (streaming line events to `writer` when
/// `stream` is set). Nothing is built here: `toplevel` is a closure already in
/// the store, which `RealiseClosure` put there.
/// and that `toplevel` is in the store with a signature by the swarm cache's
/// key, and apply it (streaming line events to `writer` when `stream` is set).
/// Nothing is built here: `toplevel` is a closure already in the store, which
/// hive-c0re pulled through the nix daemon.
///
/// `create` passes `--system-path` so `nixos-container`'s own `buildFlake()`
/// never runs; it builds to a relative `.tmp` out-link that concurrent calls
@ -228,12 +229,12 @@ pub(crate) async fn container_toplevel_action(
writer: &mut OwnedWriteHalf,
) -> Result<(String, String)> {
validate_agent_name(name)?;
let toplevel = crate::closure::applicable_for(name, toplevel)?;
let toplevel = crate::closure::applicable_for(name, toplevel).await?;
apply_toplevel(&Host::REAL, verb, name, toplevel, stream, writer).await
}
/// [`container_toplevel_action`] past the check that `toplevel` is the
/// closure realised for `name`.
/// [`container_toplevel_action`] past the check that `toplevel` is signed by
/// the swarm cache's key.
async fn apply_toplevel(
host: &Host<impl Spawn>,
verb: &str,
@ -987,11 +988,12 @@ mod host_tests {
// ---- the apply gate ----
/// Only the closure realised for the agent is applied. Nothing is
/// realised for these names, so each request is refused at the gate: the
/// message is the gate's, not a `nixos-container` failure.
/// Only a closure the swarm key signed is applied. A test has no swarm key
/// at the root path, so each request is refused at the gate: the message
/// is the gate's, not a `nixos-container` failure. The signed and unsigned
/// verdicts are tested in `closure`.
#[tokio::test]
async fn an_unrealised_or_foreign_closure_is_refused_at_the_gate() {
async fn a_closure_without_the_swarm_key_to_check_it_against_is_refused_at_the_gate() {
for verb in ["create", "update"] {
let (writer, _peer) = &mut pipe();
let err = container_toplevel_action(verb, "never-realised", TOPLEVEL, false, writer)
@ -999,7 +1001,7 @@ mod host_tests {
.unwrap_err();
let msg = format!("{err:#}");
assert!(
msg.contains("is not the closure realised for this agent (none)"),
msg.contains(&format!("never-realised: refusing {TOPLEVEL}")),
"{verb}: {msg}"
);
}

6
hive-priv/src/main.rs
View file

@ -24,8 +24,8 @@
//! [`prestart`] (the check of a container's root that runs before each start,
//! as a sub-command), [`storage`] (btrfs subvolumes, quotas, snapshots, disk
//! measurement), [`files`] (pause marker, socket dir, legacy tmpfiles),
//! [`infra`] (gateway nginx, infra containers) and [`closure`] (downloads from
//! the swarm cache). Helpers more than one family uses are in [`shared`].
//! [`infra`] (gateway nginx, infra containers) and [`closure`] (the swarm-signature
//! check before an apply, and the cache-facts refresh). Helpers more than one family uses are in [`shared`].
mod closure;
mod container;
@ -403,6 +403,8 @@ async fn exec(
ref store_path,
} => closure::realise_closure(agent, store_path).await,
PrivRequest::RefreshCacheFacts => closure::refresh_cache_facts().await,
PrivRequest::DestroyContainer { ref name } => container::destroy_container(name).await,
PrivRequest::ListContainers => container::container_run(&["list"]).await,