mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

scope the cache access to hive-core: client-side nix copy, daemon trusts only the key

Browse source
This commit is contained in:
atlas 2026-11-07 15:25:30 +01:00 • committed by mara
commit dab7768869
1 changed files with 365 additions and 271 deletions
Download patch file Download diff file Expand all files Collapse all files

365
hive-c0re/src/cache_pull.rs
View file

@ -1,31 +1,36 @@
//! Pull the swarm's closure into the store through the host's nix daemon.
//! Pull the swarm's closure into the store with `nix copy --from`.
//!
//! hive-c0re is not root and does not need to be: `nix-store --realise` of a
//! store path is a request to the daemon, which substitutes it. What the daemon
//! accepts is root's: the root unit `hive-priv-cache-facts` writes a nix.conf
//! fragment ([`CONF_FRAGMENT`]) that lists the swarm cache as a
//! `trusted-substituter`, adds its key to `trusted-public-keys` and names the
//! netrc. An unprivileged client may name a substituter only from that list,
//! cannot lower `require-sigs` or replace the keys, and a path without a
//! signature by a trusted key is refused. hive-priv then applies the toplevel
//! only if the swarm's key signed it.
//! hive-c0re is not root and does not need to be. The nix client does the HTTP
//! fetch itself, with a netrc only hive-core can read ([`NETRC`]), and hands
//! the NARs to the host's nix daemon, which stores a path from an unprivileged
//! client only if a key in its `trusted-public-keys` signed it. The root unit
//! `hive-priv-cache-facts` puts the swarm's key there through a nix.conf
//! fragment ([`CONF_FRAGMENT`]) that names no substituter and no netrc, so the
//! daemon itself never contacts the cache and no other client of it can fetch
//! from the cache or send the token. hive-priv then applies the toplevel only
//! if the swarm's key signed it.
//!
//! The cache's address is read from the fragment, so the address named is
//! byte for byte the one the daemon trusts. A missing fragment means this host
//! has not been given its cache facts; [`pull`] first asks hive-priv to run the
//! facts unit, which is how the swarm's copy of the facts reaches the host
//! without waiting for the unit's timer.
//! The cache's address is the host's configuration, handed to the unit in
//! [`URL_ENV`]. [`pull`] first asks hive-priv to run the facts unit when the
//! netrc or the key is not on the host yet, which is how the swarm's copy of the
//! facts reaches the host without waiting for the unit's timer.
use std::time::Duration;
use anyhow::Result;
use hive_priv_sock::valid_closure_path;
/// The pull token as a netrc, `root:hive-core 0640`.
pub const NETRC: &str = "/run/hyperhive-nix/netrc";
/// The nix.conf fragment `hive-priv-cache-facts` writes (root-owned, readable).
pub const CONF_FRAGMENT: &str = "/run/hyperhive-nix/cache.conf";
/// The fragment's line that lists the cache the daemon lets a client name.
const URL_SETTING: &str = "extra-trusted-substituters";
/// The unit environment variable naming the cache: `https://<domain>/<cache>`.
const URL_ENV: &str = "HIVE_C0RE_CACHE_URL";
/// The fragment's line that gives the daemon the swarm's key.
const KEY_SETTING: &str = "extra-trusted-public-keys";
/// Wall-clock bound on one pull. A stalled transfer is bounded by nix's own
/// `stalled-download-timeout`; this covers the rest.
@ -34,10 +39,10 @@ const PULL_TIMEOUT: Duration = Duration::from_mins(30);
/// Why a closure was not pulled.
#[derive(Debug, PartialEq, Eq)]
pub enum PullError {
/// This host has no cache facts: no fragment, no address in it, or a
/// daemon that has not loaded it. Waiting for the facts unit fixes it.
/// This host has no cache facts: no address, no netrc for hive-core, or no
/// key in the fragment. Waiting for the facts unit fixes the last two.
FactsMissing(String),
/// Nix refused the path: no signature by a key this host trusts.
/// The daemon refused the path: no signature by a key it trusts.
SignatureRefused(String),
/// The cache does not have the path.
NotFound(String),
@ -56,18 +61,18 @@ impl std::fmt::Display for PullError {
),
Self::SignatureRefused(last) => write!(
f,
"nix refused the closure, which carries no signature by a key this host trusts: {last}"
"the nix daemon refused the closure, which carries no signature by a key this host trusts: {last}"
),
Self::NotFound(last) => write!(f, "the swarm cache does not have the closure: {last}"),
Self::Failed(last) => f.write_str(last),
Self::TimedOut => write!(f, "nix-store timed out after {}s", PULL_TIMEOUT.as_secs()),
Self::TimedOut => write!(f, "nix copy timed out after {}s", PULL_TIMEOUT.as_secs()),
}
}
}
impl std::error::Error for PullError {}
/// What a `nix-store` run came to.
/// What a `nix copy` run came to.
pub enum Run {
Exited { success: bool, stderr: String },
TimedOut,
@ -75,68 +80,63 @@ pub enum Run {
/// What [`pull`] does on the host, so a test can stand in for it.
pub trait Effects: Sync {
/// The fragment's text; `None` when it cannot be read.
async fn fragment(&self) -> Option<String>;
/// The cache's address, when the host names one.
fn cache_url(&self) -> Option<String>;
/// Whether hive-core can read the netrc and the fragment gives the daemon
/// the swarm's key.
async fn facts_present(&self) -> bool;
/// Run the facts unit through hive-priv and wait for it.
async fn refresh(&self) -> Result<()>;
/// Run `nix-store` with `args`, bounded by [`PULL_TIMEOUT`].
async fn nix_store(&self, args: Vec<String>) -> std::io::Result<Run>;
/// Run `nix` with `args`, bounded by [`PULL_TIMEOUT`].
async fn nix(&self, args: Vec<String>) -> std::io::Result<Run>;
}
/// The cache's address in `fragment`: the value of its
/// `extra-trusted-substituters` line, when that is one https address.
/// Whether `fragment` has a non-empty `extra-trusted-public-keys` line.
#[must_use]
pub fn cache_url(fragment: &str) -> Option<&str> {
fragment.lines().find_map(|line| {
let (name, value) = line.split_once('=')?;
let value = value.trim();
(name.trim() == URL_SETTING
&& value.starts_with("https://")
&& !value.contains(char::is_whitespace))
.then_some(value)
pub fn gives_key(fragment: &str) -> bool {
fragment.lines().any(|line| {
line.split_once('=')
.is_some_and(|(name, value)| name.trim() == KEY_SETTING && !value.trim().is_empty())
})
}
/// The `nix-store` arguments that fetch `store_path` and nothing else.
/// The `nix` arguments that fetch `store_path` from the cache at `url`.
///
/// - `--option substituters <url>` names the one cache the daemon lists as
/// trusted. An unprivileged client's `substituters` that is not on that list
/// empties the list and fails the pull, so the address must be the fragment's.
/// - `--max-jobs 0` forbids a local build, so a path the cache lacks fails.
/// - No `trusted-public-keys`, `require-sigs` or `netrc-file`: the daemon
/// ignores them from an unprivileged client and applies its own.
/// - `copy --from <url>` makes the client fetch. The daemon is never asked to
/// contact the cache, so it needs no trust in the address.
/// - `--option netrc-file` is the client's own setting: it reads the file as
/// hive-core. The daemon ignores the setting from an unprivileged client.
/// - No `--no-check-sigs`: the daemon checks every path's signature against its
/// trusted keys. Nothing builds: `copy` only imports.
#[must_use]
pub fn pull_args(store_path: &str, url: &str) -> Vec<String> {
pub fn pull_args(store_path: &str, url: &str, netrc: &str) -> Vec<String> {
[
"--realise",
store_path,
"--max-jobs",
"0",
"--option",
"substituters",
"--extra-experimental-features",
"nix-command",
"copy",
"--from",
url,
"--option",
"netrc-file",
netrc,
store_path,
]
.map(str::to_owned)
.into()
}
/// Name nix's refusal from its stderr. A refusal for want of a signature
/// or a daemon that does not trust the address is told apart from a path the
/// cache lacks and from a cache that is down.
/// Name nix's refusal from its stderr. A refusal for want of a signature is
/// told apart from a path the cache lacks and from a cache that is down.
fn classify(stderr: &str) -> PullError {
let last = stderr
.lines()
.rev()
.find(|l| !l.trim().is_empty())
.unwrap_or("nix-store failed with no message")
.unwrap_or("nix copy failed with no message")
.trim()
.to_owned();
let lower = stderr.to_lowercase();
if lower.contains("untrusted substituter") {
PullError::FactsMissing(
"the nix daemon does not list the swarm cache as a trusted substituter".to_owned(),
)
} else if lower.contains("not signed by any of the keys") || lower.contains("signature") {
if lower.contains("signature") {
PullError::SignatureRefused(last)
} else if lower.contains("don't know how to build")
|| lower.contains("no substituter that can build")
@ -150,7 +150,7 @@ fn classify(stderr: &str) -> PullError {
/// Pull `store_path` into the store.
///
/// # Errors
/// [`PullError`]: no cache facts on this host (after one refresh), nix
/// [`PullError`]: no cache facts on this host (after one refresh), the daemon
/// refusing the signature, the cache lacking the path, nix failing otherwise,
/// or a timeout.
pub async fn pull(store_path: &str) -> Result<(), PullError> {
@ -163,11 +163,14 @@ async fn pull_with(fx: &impl Effects, store_path: &str) -> Result<(), PullError>
"{store_path:?} is not a store path"
)));
}
let url = address_after_refresh(fx).await?;
let url = fx
.cache_url()
.ok_or_else(|| PullError::FactsMissing("this host names no cache".to_owned()))?;
ensure_facts(fx).await?;
match fx
.nix_store(pull_args(store_path, &url))
.nix(pull_args(store_path, &url, NETRC))
.await
.map_err(|e| PullError::Failed(format!("running nix-store: {e}")))?
.map_err(|e| PullError::Failed(format!("running nix: {e}")))?
{
Run::Exited { success: true, .. } => Ok(()),
Run::Exited { stderr, .. } => Err(classify(&stderr)),
@ -175,43 +178,44 @@ async fn pull_with(fx: &impl Effects, store_path: &str) -> Result<(), PullError>
}
}
/// The cache's address, after one facts refresh when the host has none.
async fn address_after_refresh(fx: &impl Effects) -> Result<String, PullError> {
if let Some(url) = address(fx).await {
return Ok(url);
/// The facts are on the host, after one refresh when they are not.
async fn ensure_facts(fx: &impl Effects) -> Result<(), PullError> {
if fx.facts_present().await {
return Ok(());
}
let refreshed = fx.refresh().await;
address(fx).await.ok_or_else(|| {
PullError::FactsMissing(match refreshed {
Ok(()) => "the facts unit ran and the swarm has given this hive none".to_owned(),
Err(e) => format!("refreshing them failed: {e:#}"),
})
})
if fx.facts_present().await {
return Ok(());
}
Err(PullError::FactsMissing(match refreshed {
Ok(()) => "the facts unit ran and the swarm has given this hive none".to_owned(),
Err(e) => format!("refreshing them failed: {e:#}"),
}))
}
async fn address(fx: &impl Effects) -> Option<String> {
fx.fragment()
.await
.as_deref()
.and_then(cache_url)
.map(str::to_owned)
}
/// The host: the fragment on disk, hive-priv for the refresh, `nix-store` from
/// the system profile through the daemon.
/// The host: the environment and files on disk, hive-priv for the refresh,
/// `nix` from the system profile.
struct Real;
impl Effects for Real {
async fn fragment(&self) -> Option<String> {
tokio::fs::read_to_string(CONF_FRAGMENT).await.ok()
fn cache_url(&self) -> Option<String> {
std::env::var(URL_ENV)
.ok()
.map(|u| u.trim().to_owned())
.filter(|u| u.starts_with("https://") && !u.contains(char::is_whitespace))
}
async fn facts_present(&self) -> bool {
let fragment = tokio::fs::read_to_string(CONF_FRAGMENT).await;
fragment.is_ok_and(|f| gives_key(&f)) && tokio::fs::File::open(NETRC).await.is_ok()
}
async fn refresh(&self) -> Result<()> {
crate::priv_client::refresh_cache_facts().await
}
async fn nix_store(&self, args: Vec<String>) -> std::io::Result<Run> {
let mut cmd = tokio::process::Command::new("nix-store");
async fn nix(&self, args: Vec<String>) -> std::io::Result<Run> {
let mut cmd = tokio::process::Command::new("nix");
// Explicit, so the store is the daemon's whatever the environment.
cmd.args(args)
.env("NIX_REMOTE", "daemon")
@ -240,27 +244,23 @@ mod tests {
const PATH: &str = "/nix/store/0123456789abcdfghijklmnpqrsvwxyz-nixos-system-atlas";
const URL: &str = "https://cache.example/swarm";
fn fragment(url: &str) -> String {
format!(
"extra-trusted-substituters = {url}\nextra-trusted-public-keys = swarm:AAAA\nnetrc-file = /run/hyperhive-priv/attic/netrc"
)
}
/// A host that answers from a script: the fragment as it is now (and the
/// one a refresh leaves), what `nix-store` comes to, and a log of calls.
/// A host that answers from a script: whether the facts are present now
/// (and after a refresh), what `nix` comes to, and a log of calls.
struct Fake {
fragment: Mutex<Option<String>>,
after_refresh: Option<String>,
url: Option<&'static str>,
present: Mutex<bool>,
present_after_refresh: bool,
refresh_fails: bool,
nix: Mutex<Option<Run>>,
calls: Mutex<Vec<String>>,
}
impl Fake {
fn new(fragment: Option<String>, nix: Run) -> Self {
fn new(present: bool, nix: Run) -> Self {
Self {
fragment: Mutex::new(fragment),
after_refresh: None,
url: Some(URL),
present: Mutex::new(present),
present_after_refresh: true,
refresh_fails: false,
nix: Mutex::new(Some(nix)),
calls: Mutex::new(Vec::new()),
@ -273,9 +273,13 @@ mod tests {
}
impl Effects for Fake {
async fn fragment(&self) -> Option<String> {
self.calls.lock().unwrap().push("fragment".to_owned());
self.fragment.lock().unwrap().clone()
fn cache_url(&self) -> Option<String> {
self.url.map(str::to_owned)
}
async fn facts_present(&self) -> bool {
self.calls.lock().unwrap().push("facts".to_owned());
*self.present.lock().unwrap()
}
async fn refresh(&self) -> Result<()> {
@ -283,15 +287,15 @@ mod tests {
if self.refresh_fails {
anyhow::bail!("hive-priv unreachable");
}
*self.fragment.lock().unwrap() = self.after_refresh.clone();
*self.present.lock().unwrap() = self.present_after_refresh;
Ok(())
}
async fn nix_store(&self, args: Vec<String>) -> std::io::Result<Run> {
async fn nix(&self, args: Vec<String>) -> std::io::Result<Run> {
self.calls
.lock()
.unwrap()
.push(format!("nix-store {}", args.join(" ")));
.push(format!("nix {}", args.join(" ")));
Ok(self.nix.lock().unwrap().take().expect("one run"))
}
}
@ -303,87 +307,106 @@ mod tests {
}
}
fn option<'a>(args: &'a [String], name: &str) -> Option<&'a str> {
args.windows(3)
.find(|w| w[0] == "--option" && w[1] == name)
.map(|w| w[2].as_str())
fn after(args: &[String], flag: &str) -> Option<String> {
args.iter()
.position(|a| a == flag)
.and_then(|i| args.get(i + 1))
.cloned()
}
/// An unprivileged client can name only the trusted address and cannot set
/// keys, signatures or the netrc, so the command carries exactly that, and
/// never lets nix build.
/// The client fetches: `copy --from <url>`, with its own netrc as a client
/// setting. Nothing asks the daemon to contact the cache or to skip the
/// signature check, and nothing can build.
#[test]
fn the_command_names_only_the_swarm_cache_and_never_builds() {
let args = pull_args(PATH, URL);
assert_eq!(args[..2], ["--realise", PATH]);
let jobs = args.iter().position(|a| a == "--max-jobs").expect("flag");
assert_eq!(args[jobs + 1], "0");
assert_eq!(option(&args, "substituters"), Some(URL));
let options: Vec<_> = args.iter().filter(|a| *a == "--option").collect();
assert_eq!(options.len(), 1, "{args:?}");
for forbidden in ["trusted-public-keys", "require-sigs", "netrc-file"] {
assert_eq!(option(&args, forbidden), None, "{forbidden}");
fn the_command_copies_from_the_cache_with_the_clients_netrc_and_checks_signatures() {
let args = pull_args(PATH, URL, NETRC);
assert_eq!(args.last().map(String::as_str), Some(PATH));
let at = args.iter().position(|a| a == "copy").expect("verb");
assert_eq!(args[at + 1], "--from");
assert_eq!(args[at + 2], URL);
assert_eq!(after(&args, "--option").as_deref(), Some("netrc-file"));
assert_eq!(
args.windows(3)
.find(|w| w[0] == "--option" && w[1] == "netrc-file")
.map(|w| w[2].as_str()),
Some(NETRC)
);
// Exactly one setting, and none that widens trust.
assert_eq!(
args.iter().filter(|a| *a == "--option").count(),
1,
"{args:?}"
);
for forbidden in [
"--no-check-sigs",
"--substitute-on-destination",
"--to",
"substituters",
"trusted-public-keys",
"require-sigs",
] {
assert!(
!args.iter().any(|a| a == forbidden),
"{forbidden}: {args:?}"
);
}
assert!(!args.iter().any(|a| a.starts_with("extra-")), "{args:?}");
}
#[test]
fn the_address_is_the_fragments_https_line() {
assert_eq!(cache_url(&fragment(URL)), Some(URL));
fn the_fragment_gives_the_key_when_it_has_a_value() {
assert!(gives_key("extra-trusted-public-keys = swarm:AAAA"));
for bad in [
String::new(),
"netrc-file = /x".to_owned(),
fragment("http://cache.example/swarm"),
fragment("https://cache.example/a b"),
fragment(""),
// The key's line is not the address's.
"extra-trusted-public-keys = https://cache.example/swarm".to_owned(),
"",
"extra-trusted-public-keys =",
"extra-trusted-public-keys",
"trusted-public-keys = swarm:AAAA",
"extra-trusted-substituters = https://cache.example/swarm",
] {
assert_eq!(cache_url(&bad), None, "{bad:?}");
assert!(!gives_key(bad), "{bad:?}");
}
}
#[tokio::test]
async fn with_the_fragment_in_place_it_pulls_without_a_refresh() {
let fx = Fake::new(Some(fragment(URL)), exited(true, ""));
async fn with_the_facts_in_place_it_pulls_without_a_refresh() {
let fx = Fake::new(true, exited(true, ""));
pull_with(&fx, PATH).await.expect("pulled");
assert_eq!(
fx.calls(),
[
"fragment".to_owned(),
format!("nix-store {}", pull_args(PATH, URL).join(" "))
"facts".to_owned(),
format!("nix {}", pull_args(PATH, URL, NETRC).join(" "))
]
);
}
/// Missing facts are refreshed once, and the pull then goes ahead with the
/// address the refresh wrote.
/// Missing facts are refreshed once, and the pull then goes ahead.
#[tokio::test]
async fn a_missing_fragment_is_refreshed_before_the_pull() {
let mut fx = Fake::new(None, exited(true, ""));
fx.after_refresh = Some(fragment(URL));
async fn missing_facts_are_refreshed_before_the_pull() {
let fx = Fake::new(false, exited(true, ""));
pull_with(&fx, PATH).await.expect("pulled");
assert_eq!(
fx.calls(),
[
"fragment".to_owned(),
"facts".to_owned(),
"refresh".to_owned(),
"fragment".to_owned(),
format!("nix-store {}", pull_args(PATH, URL).join(" "))
"facts".to_owned(),
format!("nix {}", pull_args(PATH, URL, NETRC).join(" "))
]
);
}
/// Facts that are still absent after the refresh, whether the unit ran or
/// hive-priv could not start it, stop the pull before nix is asked.
/// Facts still absent after the refresh, whether the unit ran or hive-priv
/// could not start it, stop the pull before nix is asked.
#[tokio::test]
async fn facts_still_missing_after_the_refresh_are_reported_and_nix_is_not_run() {
for (case, refresh_fails, want) in [
("unit ran", false, "none"),
("refresh failed", true, "hive-priv unreachable"),
] {
let mut fx = Fake::new(None, exited(true, ""));
let mut fx = Fake::new(false, exited(true, ""));
fx.refresh_fails = refresh_fails;
fx.present_after_refresh = false;
let err = pull_with(&fx, PATH).await.expect_err(case);
let PullError::FactsMissing(why) = &err else {
panic!("{case}: {err:?}");
@ -393,44 +416,39 @@ mod tests {
format!("{err}").contains("a later deploy retries"),
"{case}"
);
assert!(
!fx.calls().iter().any(|c| c.starts_with("nix-store")),
"{case}"
);
assert!(!fx.calls().iter().any(|c| c.starts_with("nix")), "{case}");
}
}
/// A host that names no cache has nothing to refresh toward.
#[tokio::test]
async fn a_fragment_without_an_address_is_refreshed_like_a_missing_one() {
let mut fx = Fake::new(Some("netrc-file = /x".to_owned()), exited(true, ""));
fx.after_refresh = Some(fragment(URL));
pull_with(&fx, PATH).await.expect("pulled");
assert!(fx.calls().contains(&"refresh".to_owned()));
async fn a_host_that_names_no_cache_does_not_pull_or_refresh() {
let mut fx = Fake::new(true, exited(true, ""));
fx.url = None;
let err = pull_with(&fx, PATH).await.expect_err("no cache");
assert!(matches!(err, PullError::FactsMissing(_)), "{err:?}");
assert!(fx.calls().is_empty(), "{:?}", fx.calls());
}
#[test]
fn nixs_refusals_are_told_apart() {
for (stderr, want) in [
(
"warning: ignoring substitute for '/nix/store/x' from 'https://c', as it's not signed by any of the keys in 'trusted-public-keys'\nerror: build of '/nix/store/x' failed",
"error: cannot add path '/nix/store/x' because it lacks a signature by a trusted key",
"signature",
),
(
"error: cannot add path 'x' because it lacks a signature by a trusted key",
"signature",
),
(
"warning: ignoring untrusted substituter 'https://c', you are not a trusted user.\ndon't know how to build these paths:\nerror: build of 'x' failed",
"facts",
),
(
"don't know how to build these paths:\n /nix/store/x\nerror: build of '/nix/store/x' failed",
"don't know how to build these paths:\n /nix/store/x\nerror: path '/nix/store/x' is required, but there is no substituter that can build it",
"not found",
),
(
"error: path '/nix/store/x' is required, but there is no substituter that can build it",
"not found",
),
(
"error: unable to download 'https://c/nix-cache-info': HTTP error 401",
"other",
),
(
"error: unable to download 'https://c/x.narinfo': Couldn't connect to server",
"other",
@ -438,10 +456,9 @@ mod tests {
] {
let kind = match classify(stderr) {
PullError::SignatureRefused(_) => "signature",
PullError::FactsMissing(_) => "facts",
PullError::NotFound(_) => "not found",
PullError::Failed(_) => "other",
PullError::TimedOut => "timeout",
PullError::FactsMissing(_) | PullError::TimedOut => "unexpected",
};
assert_eq!(kind, want, "{stderr}");
}
@ -455,17 +472,17 @@ mod tests {
#[tokio::test]
async fn a_failed_run_is_mapped_and_a_timeout_is_a_timeout() {
let fx = Fake::new(
Some(fragment(URL)),
true,
exited(
false,
"warning: ignoring substitute from 'https://c', as it's not signed by any of the keys",
"error: cannot add path 'x' because it lacks a signature by a trusted key",
),
);
assert!(matches!(
pull_with(&fx, PATH).await,
Err(PullError::SignatureRefused(_))
));
let fx = Fake::new(Some(fragment(URL)), Run::TimedOut);
let fx = Fake::new(true, Run::TimedOut);
assert_eq!(pull_with(&fx, PATH).await, Err(PullError::TimedOut));
}
@ -473,7 +490,7 @@ mod tests {
#[tokio::test]
async fn only_a_store_path_is_pulled() {
for bad in ["", "main", "/etc/passwd", "--help", "/nix/store/x"] {
let fx = Fake::new(Some(fragment(URL)), exited(true, ""));
let fx = Fake::new(true, exited(true, ""));
assert!(
matches!(pull_with(&fx, bad).await, Err(PullError::Failed(_))),
"{bad}"