vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

run the root check warn-only on hyperhive's own containers, per unit, not on the container@ template

Browse source
This commit is contained in:
atlas 2026-10-28 23:37:59 +01:00
commit 24afc89ab3
1 changed files with 707 additions and 172 deletions
Download patch file Download diff file Expand all files Collapse all files

440
hive-priv/src/prestart.rs
View file

@ -3,30 +3,44 @@
//! nixpkgs' `container@.service` start script runs as host root against the
//! container's own root, which is persistent and writable by the container.
//! `mkdir -p`, `chmod`, `touch` and `cp --remove-destination` there follow a
//! link the container planted at any of the names the script touches. A
//! drop-in of the `container@` template
//! (`nix/host-modules/container-rootfs-check.nix`) runs
//! `hive-priv check-rootfs %i` as an `ExecStartPre=` ahead of that script for
//! every instance; a non-zero exit fails the start. The container is stopped
//! link the container planted at any of the names the script touches. An agent
//! container's drop-in ([`crate::container_config`]) puts
//! `hive-priv check-rootfs <container>` ahead of that script as an
//! `ExecStartPre=`; a non-zero exit fails the start. The container is stopped
//! while this runs, so nothing changes under the walk.
//!
//! Hyperhive's own declarative containers (the forge, the matrix server, the
//! swarm services) run it as `check-rootfs --warn-only <container>` from their
//! own unit (`nix/host-modules/lib/container-rootfs-check.nix`): each violation
//! is logged at warning level and the start goes on.
//!
//! Every component is looked at with `symlink_metadata`: a link is reported,
//! never followed. Nothing is deleted or repaired, so whatever is there stays
//! for the operator.
//!
//! The drop-in names the running hive-priv binary, which nothing else roots
//! once the system has moved on to a newer build. [`pin_binary`] keeps a GC
//! root on it for as long as the drop-in exists, and [`unpin_binary`] drops
//! the root with the drop-in.
//!
//! The entry point is [`run`]; [`check`] holds the rules.
use std::fs::{FileType, Metadata};
use std::os::unix::fs::{DirBuilderExt as _, symlink};
use std::path::{Path, PathBuf};
use anyhow::{Context as _, Result, bail};
use crate::container_config::container_rootfs;
use crate::shared::validate_name_chars;
use crate::container_config::{container_rootfs, validate_container_system_name};
use crate::shared::{partial_name, validate_name_chars};
/// The sub-command the template drop-in's `ExecStartPre=` names.
/// The sub-command the drop-in's `ExecStartPre=` names.
pub(crate) const SUBCOMMAND: &str = "check-rootfs";
/// The flag that turns a violation into a warning: the check runs in full and
/// exits 0.
const WARN_ONLY: &str = "--warn-only";
/// The per-container nix profile directories on the host.
const PROFILES_ROOT: &str = "/nix/var/nix/profiles/per-container";
@ -59,47 +73,157 @@ const REGULAR_ONLY: [&str; 2] = ["etc/machine-id", "etc/resolv.conf"];
/// `/etc/static` once it has booted.
const OS_RELEASE: &str = "etc/os-release";
/// Where hive-priv roots the build named by each container's `ExecStartPre=`:
/// `<container>` is a symlink to that build's store path. The directory is
/// under the host's `/nix/var/nix/gcroots`, not `per-container`, which is the
/// only part of it a container is given (`--bind=.../gcroots/per-container/<c>`
/// in the nixpkgs start script), so a container can't reach it.
const GCROOTS: &str = "/nix/var/nix/gcroots/hyperhive-check-rootfs";
/// Root the store path of `exe` for `container`, replacing the root a previous
/// write left, so the build a drop-in names outlives a garbage collection for
/// as long as the drop-in does. One link per container, so they don't pile up.
pub(crate) fn pin_binary(container: &str, exe: &Path) -> Result<()> {
pin_binary_in(Path::new(GCROOTS), Path::new(STORE), container, exe)
}
/// Drop the root [`pin_binary`] left for `container`, once its drop-in is gone.
/// A name that holds anything but a link is left alone and reported.
pub(crate) fn unpin_binary(container: &str) -> Result<()> {
unpin_binary_in(Path::new(GCROOTS), container)
}
/// [`pin_binary`] with the roots directory and the store as parameters.
///
/// A binary outside `store` can't be collected, so it gets no root. The link
/// is made under a temporary name and renamed over `<container>`: `rename`
/// replaces whatever link is there without following it.
fn pin_binary_in(roots: &Path, store: &Path, container: &str, exe: &Path) -> Result<()> {
let Some(entry) = exe
.strip_prefix(store)
.ok()
.and_then(|rest| rest.components().next())
else {
return Ok(());
};
let target = store.join(entry);
std::fs::DirBuilder::new()
.recursive(true)
.mode(0o755)
.create(roots)
.with_context(|| format!("create {}", roots.display()))?;
if !lstat(roots)?.is_some_and(|m| m.file_type().is_dir()) {
bail!("{} is not a real directory", roots.display());
}
let tmp = roots.join(partial_name(container));
symlink(&target, &tmp).with_context(|| format!("create {}", tmp.display()))?;
let root = roots.join(container);
std::fs::rename(&tmp, &root).map_err(|e| {
// Best effort: a leftover temp is a root too, but a harmless one.
let _ = std::fs::remove_file(&tmp);
anyhow::Error::new(e).context(format!("replace {}", root.display()))
})
}
/// [`unpin_binary`] with the roots directory as a parameter.
fn unpin_binary_in(roots: &Path, container: &str) -> Result<()> {
let root = roots.join(container);
match lstat(&root)? {
None => Ok(()),
Some(meta) if meta.file_type().is_symlink() => {
std::fs::remove_file(&root).with_context(|| format!("remove {}", root.display()))
}
Some(meta) => bail!(
"{} is {}, not a link: left alone",
root.display(),
kind(meta.file_type())
),
}
}
/// Handle `hive-priv <subcommand> ...` when the first argument names one.
/// `None` means it is not a sub-command and the daemon starts.
///
/// Prints one line to stderr and returns exit code 1 on a violation, so the
/// reason lands in the journal next to the failed unit.
/// `check-rootfs <container>` is strict: it prints one line to stderr and
/// returns exit code 1 on a violation, so the reason lands in the journal next
/// to the failed unit. `check-rootfs --warn-only <container>` prints every
/// violation as a warning and returns 0.
pub(crate) fn run(args: &[String]) -> Option<i32> {
if args.first().map(String::as_str) != Some(SUBCOMMAND) {
return None;
}
let result = match args {
[_, container] => check_container(container),
_ => Err(anyhow::anyhow!("usage: hive-priv {SUBCOMMAND} <container>")),
};
Some(match result {
Ok(()) => 0,
Err(e) => {
eprintln!("hive-priv {SUBCOMMAND}: refusing to start: {e:#}");
1
let (container, warn_only) = match args {
[_, container] => (container, false),
[_, flag, container] if flag == WARN_ONLY => (container, true),
_ => {
eprintln!("usage: hive-priv {SUBCOMMAND} [{WARN_ONLY}] <container>");
return Some(1);
}
})
};
let found = found_in_container(container, warn_only);
let (code, lines) = report(container, &found, warn_only);
for line in lines {
eprintln!("{line}");
}
Some(code)
}
/// [`check`] on the host paths of `container`.
fn check_container(container: &str) -> Result<()> {
let (root, system) = container_paths(container)?;
check(&root, &system, Path::new(STORE))
/// What a start would be refused for, on the host paths of `container`.
///
/// Strict mode stops at the first violation, as the start would. Warn-only
/// lists them all, and counts a name it can't use as one more: it never fails
/// the start over its own input.
fn found_in_container(container: &str, warn_only: bool) -> Vec<String> {
let paths = container_paths(container, warn_only);
match (paths, warn_only) {
(Ok((root, system)), true) => violations(&root, &system, Path::new(STORE)),
(Ok((root, system)), false) => check(&root, &system, Path::new(STORE))
.err()
.map(|e| format!("{e:#}"))
.into_iter()
.collect(),
(Err(e), _) => vec![format!("{e:#}")],
}
}
/// The container's root and its profile's `system` link on the host.
///
/// The unit instance supplies the name and it selects both paths, so anything
/// but a plain container name is refused: lowercase ascii, digits and
/// hyphens. That holds no `/` and is never `.` or `..`.
fn container_paths(container: &str) -> Result<(PathBuf, PathBuf)> {
validate_name_chars(container)?;
/// An agent container is named `h-<agent>`. A warn-only check also takes the
/// platform containers' names, which are lowercase ascii, digits and hyphens.
/// Either way the name holds no `/` and is never `.` or `..`.
fn container_paths(container: &str, warn_only: bool) -> Result<(PathBuf, PathBuf)> {
if warn_only {
validate_name_chars(container)?;
} else {
validate_container_system_name(container)?;
}
Ok((
container_rootfs(container),
Path::new(PROFILES_ROOT).join(container).join("system"),
))
}
/// The exit code and the stderr lines for the violations `found`.
///
/// Warn-only lines carry the `<4>` prefix, which journald reads as the
/// warning priority, so a log query can select them by severity.
fn report(container: &str, found: &[String], warn_only: bool) -> (i32, Vec<String>) {
if warn_only {
let lines = found
.iter()
.map(|v| format!("<4>rootfs check (warn-only): {container}: {v}"))
.collect();
return (0, lines);
}
match found.first() {
Some(v) => (
1,
vec![format!("hive-priv {SUBCOMMAND}: refusing to start: {v}")],
),
None => (0, Vec::new()),
}
}
/// Check the paths the start script touches as root, returning an error that
/// names the first path to hold something it must not.
///
@ -109,14 +233,36 @@ fn container_paths(container: &str) -> Result<(PathBuf, PathBuf)> {
/// A path that does not exist passes: a first start has nothing yet, and the
/// script creates it.
fn check(root: &Path, system: &Path, store: &Path) -> Result<()> {
match violations(root, system, store).into_iter().next() {
None => Ok(()),
Some(first) => bail!("{first}"),
}
}
/// Every path under the rules of [`check`] that holds something it must not,
/// in the order [`check`] looks at them, each message once.
fn violations(root: &Path, system: &Path, store: &Path) -> Vec<String> {
let mut found = Vec::new();
for dir in DIRS {
check_dir(root, dir)?;
note(&mut found, check_dir(root, dir));
}
for file in REGULAR_ONLY {
check_file(root, file, false)?;
note(&mut found, check_file(root, file, false));
}
note(&mut found, check_file(root, OS_RELEASE, true));
note(&mut found, check_system(system, store));
found
}
/// Add the message of an error to `found`, unless it is already there: a link
/// at `usr` is the finding for `usr/bin` too.
fn note(found: &mut Vec<String>, result: Result<()>) {
if let Err(e) = result {
let message = format!("{e:#}");
if !found.contains(&message) {
found.push(message);
}
}
check_file(root, OS_RELEASE, true)?;
check_system(system, store)
}
/// `symlink_metadata`, with `None` for a path that does not exist. The one
@ -221,8 +367,8 @@ mod tests {
use std::os::unix::fs::symlink;
use std::path::PathBuf;
use super::{check, container_paths, run};
use crate::shared::test_support::scratch;
use super::{check, container_paths, pin_binary_in, report, run, unpin_binary_in, violations};
use crate::shared::test_support::{entries, scratch};
/// A scratch container: root, the profile link's path and a store with one
/// toplevel in it. Nothing is linked yet.
@ -450,11 +596,133 @@ mod tests {
assert!(target.is_dir());
}
/// A scratch store holding one build with a binary in it, a roots dir and
/// the binary's path.
fn pin_fixture() -> (PathBuf, PathBuf, PathBuf, PathBuf) {
let base = scratch();
let store = base.join("store");
let build = store.join("abc-hive-priv");
std::fs::create_dir_all(build.join("bin")).unwrap();
let exe = build.join("bin/hive-priv");
std::fs::write(&exe, "").unwrap();
(base.join("roots"), store, build, exe)
}
#[test]
fn a_plain_container_name_selects_its_root_and_profile() {
// Agent and non-agent names alike.
for name in ["h-iris", "hive-forge", "swarm-review-classifier"] {
let (root, system) = container_paths(name).unwrap();
fn the_root_points_at_the_store_path_the_drop_in_names() {
let (roots, store, build, exe) = pin_fixture();
pin_binary_in(&roots, &store, "h-iris", &exe).unwrap();
let root = roots.join("h-iris");
assert!(std::fs::symlink_metadata(&root).unwrap().is_symlink());
// The build, not the binary inside it.
assert_eq!(std::fs::read_link(&root).unwrap(), build);
assert_eq!(std::fs::read_dir(&roots).unwrap().count(), 1);
}
#[test]
fn a_rewrite_replaces_the_root_and_leaves_no_temp() {
let (roots, store, build, exe) = pin_fixture();
let newer = store.join("def-hive-priv");
std::fs::create_dir_all(newer.join("bin")).unwrap();
pin_binary_in(&roots, &store, "h-iris", &exe).unwrap();
pin_binary_in(&roots, &store, "h-iris", &newer.join("bin/hive-priv")).unwrap();
pin_binary_in(&roots, &store, "h-bob", &exe).unwrap();
assert_eq!(std::fs::read_link(roots.join("h-iris")).unwrap(), newer);
assert_eq!(std::fs::read_link(roots.join("h-bob")).unwrap(), build);
assert_eq!(entries(&roots), ["h-bob", "h-iris"]);
}
#[test]
fn a_link_planted_at_the_root_name_is_replaced_not_followed() {
let (roots, store, build, exe) = pin_fixture();
let victim = victim();
std::fs::write(victim.join("keep"), "data").unwrap();
std::fs::create_dir_all(&roots).unwrap();
symlink(&victim, roots.join("h-iris")).unwrap();
pin_binary_in(&roots, &store, "h-iris", &exe).unwrap();
assert_eq!(std::fs::read_link(roots.join("h-iris")).unwrap(), build);
assert_eq!(
std::fs::read_to_string(victim.join("keep")).unwrap(),
"data"
);
assert_eq!(std::fs::read_dir(&victim).unwrap().count(), 1);
}
#[test]
fn a_roots_dir_that_is_a_link_is_refused() {
let (roots, store, _, exe) = pin_fixture();
let victim = victim();
symlink(&victim, &roots).unwrap();
let msg = format!(
"{:#}",
pin_binary_in(&roots, &store, "h-iris", &exe).unwrap_err()
);
assert!(msg.contains("is not a real directory"), "{msg}");
assert_eq!(std::fs::read_dir(&victim).unwrap().count(), 0);
}
#[test]
fn a_binary_outside_the_store_gets_no_root() {
let (roots, store, _, _) = pin_fixture();
let elsewhere = scratch().join("hive-priv");
pin_binary_in(&roots, &store, "h-iris", &elsewhere).unwrap();
assert!(!roots.exists());
}
#[test]
fn unpinning_removes_the_container_root_and_nothing_else() {
let (roots, store, build, exe) = pin_fixture();
pin_binary_in(&roots, &store, "h-iris", &exe).unwrap();
pin_binary_in(&roots, &store, "h-bob", &exe).unwrap();
unpin_binary_in(&roots, "h-iris").unwrap();
assert_eq!(entries(&roots), ["h-bob"]);
// Already gone: fine. The build it named is untouched.
unpin_binary_in(&roots, "h-iris").unwrap();
assert!(build.join("bin/hive-priv").is_file());
}
#[test]
fn unpinning_leaves_a_name_that_is_not_a_link_alone() {
let (roots, _, _, _) = pin_fixture();
std::fs::create_dir_all(roots.join("h-iris")).unwrap();
std::fs::write(roots.join("h-iris/keep"), "").unwrap();
let msg = format!("{:#}", unpin_binary_in(&roots, "h-iris").unwrap_err());
assert!(msg.contains("left alone"), "{msg}");
assert!(roots.join("h-iris/keep").exists());
}
#[test]
fn only_the_check_rootfs_sub_command_is_taken() {
let args = |a: &[&str]| a.iter().map(|s| (*s).to_owned()).collect::<Vec<_>>();
assert_eq!(run(&[]), None);
assert_eq!(run(&args(&["serve"])), None);
// Wrong arity, an unknown flag and a name that is not an agent
// container are refused, whatever is on disk.
assert_eq!(run(&args(&["check-rootfs"])), Some(1));
assert_eq!(run(&args(&["check-rootfs", "not-hive"])), Some(1));
assert_eq!(run(&args(&["check-rootfs", "--strict", "h-iris"])), Some(1));
assert_eq!(run(&args(&["check-rootfs", "--warn-only"])), Some(1));
}
#[test]
fn a_warn_only_run_exits_zero_even_for_a_name_it_cannot_use() {
let args = |a: &[&str]| a.iter().map(|s| (*s).to_owned()).collect::<Vec<_>>();
assert_eq!(
run(&args(&["check-rootfs", "--warn-only", "no-such-container"])),
Some(0)
);
assert_eq!(
run(&args(&["check-rootfs", "--warn-only", "../etc"])),
Some(0)
);
}
#[test]
fn strict_takes_agent_names_and_warn_only_takes_plain_ones() {
assert!(container_paths("h-iris", false).is_ok());
assert!(container_paths("hive-forge", false).is_err());
for name in ["hive-forge", "swarm-review-classifier", "h-iris"] {
let (root, system) = container_paths(name, true).unwrap();
assert_eq!(
root,
PathBuf::from(format!("/var/lib/nixos-containers/{name}"))
@ -464,10 +732,6 @@ mod tests {
PathBuf::from(format!("/nix/var/nix/profiles/per-container/{name}/system"))
);
}
}
#[test]
fn a_name_that_is_not_a_plain_container_name_is_refused() {
for name in [
"",
".",
@ -475,28 +739,84 @@ mod tests {
"../h-iris",
"a/b",
"/etc",
"h-iris/",
"H-iris",
"a.b",
"a b",
] {
assert!(container_paths(name).is_err(), "{name:?} was accepted");
}
}
#[test]
fn only_the_check_rootfs_sub_command_is_taken() {
assert_eq!(run(&[]), None);
assert_eq!(run(&["serve".to_string()]), None);
// Wrong arity and a name that selects a path outside the containers
// directory are refused, whatever is on disk.
assert_eq!(run(&["check-rootfs".to_string()]), Some(1));
for name in ["", "..", "a/b"] {
assert_eq!(
run(&["check-rootfs".to_string(), name.to_string()]),
Some(1),
"{name:?}"
assert!(
container_paths(name, true).is_err(),
"{name:?} was accepted"
);
assert!(
container_paths(name, false).is_err(),
"{name:?} was accepted"
);
}
}
/// A clean root with a link at `root` and another at `etc/resolv.conf`.
fn two_violations() -> Fixture {
let f = Fixture::clean();
std::fs::remove_dir_all(f.root.join("root")).unwrap();
symlink(victim(), f.root.join("root")).unwrap();
std::fs::remove_file(f.root.join("etc/resolv.conf")).unwrap();
symlink("/etc/hosts", f.root.join("etc/resolv.conf")).unwrap();
f
}
fn found(f: &Fixture) -> Vec<String> {
violations(&f.root, &f.system, &f.store)
}
#[test]
fn warn_only_logs_every_violation_at_warning_level_and_exits_zero() {
let f = two_violations();
let found = found(&f);
assert_eq!(found.len(), 2, "{found:?}");
let (code, lines) = report("hive-forge", &found, true);
assert_eq!(code, 0);
assert_eq!(lines.len(), 2, "{lines:?}");
for line in &lines {
assert!(
line.starts_with("<4>rootfs check (warn-only): hive-forge: "),
"{line}"
);
}
assert!(lines[0].contains("root is a symlink"), "{lines:?}");
assert!(lines[1].contains("resolv.conf is a symlink"), "{lines:?}");
}
#[test]
fn strict_refuses_with_the_first_violation_only() {
let f = two_violations();
let found = found(&f);
let (code, lines) = report("h-iris", &found, false);
assert_eq!(code, 1);
assert_eq!(lines.len(), 1, "{lines:?}");
assert!(
lines[0].starts_with("hive-priv check-rootfs: refusing to start: "),
"{lines:?}"
);
assert!(lines[0].contains("root is a symlink"), "{lines:?}");
// The rules are the same: the strict refusal is `check`'s first error.
assert_eq!(f.refused(), found[0]);
}
#[test]
fn a_clean_root_prints_nothing_in_either_mode() {
let f = Fixture::clean();
assert!(found(&f).is_empty());
assert_eq!(report("hive-forge", &[], true), (0, Vec::new()));
assert_eq!(report("h-iris", &[], false), (0, Vec::new()));
}
#[test]
fn one_link_is_reported_once_though_two_paths_run_through_it() {
let f = Fixture::clean();
std::fs::remove_dir_all(f.root.join("usr")).unwrap();
symlink(victim(), f.root.join("usr")).unwrap();
let found = found(&f);
assert_eq!(found.len(), 1, "{found:?}");
assert!(found[0].contains("usr is a symlink"), "{found:?}");
}
}