diff --git a/docs/agent-lifecycle/persistence.md b/docs/agent-lifecycle/persistence.md index 75392956..294a30a8 100644 --- a/docs/agent-lifecycle/persistence.md +++ b/docs/agent-lifecycle/persistence.md @@ -385,32 +385,46 @@ never data. **Checked before every start.** nixpkgs' start script runs as host root on the root template, and its `mkdir -p`, `chmod`, `touch` and `cp --remove-destination` follow a link the container planted at the paths -they touch. The host's nix config therefore adds a drop-in to the `container@` -template (`container@.service.d/`, from -`nix/host-modules/container-rootfs-check.nix`) with an `ExecStartPre=` that -runs `hive-priv check-rootfs %i`. It applies to every container instance, the -agents and the swarm's service containers alike, and runs on systemd's own -restarts too. systemd reads a drop-in after the unit file and runs the -`ExecStartPre=` lines in that order, all of them before `ExecStart=`, so the -check runs before the start script touches the root. The unit file names the -`hive-priv` store path, so the system closure keeps that build alive. - -The check looks at each path with `lstat`, never following a link, and fails -the start with one line on stderr naming the path and its type when: +they touch. The container's limits drop-in +(`container@h-.service.d/hyperhive-limits.conf`) therefore carries an +`ExecStartPre=` that runs `hive-priv check-rootfs h-` ahead of that +script, on systemd's own restarts too. It looks at each path with `lstat`, +never following a link, and fails the start with one line on stderr naming the +path and its type when: - `usr`, `usr/bin`, `etc`, `var`, `var/lib`, `var/lib/private` or `root` exists and isn't a real directory; - `etc/machine-id` or `etc/resolv.conf` exists and isn't a regular file (`etc/os-release` may also be a link, which the script skips and a booted NixOS guest creates); -- the profile's `system` (`/nix/var/nix/profiles/per-container//system`) +- the profile's `system` (`/nix/var/nix/profiles/per-container/h-/system`) exists and isn't a link that resolves into `/nix/store`, or the `system` file inside the resolved directory isn't a regular file. -The name `%i` selects both paths, so the check refuses a name that isn't -lowercase ascii, digits and hyphens. Absent paths pass, so a first start works. -The check neither deletes nor repairs: the operator removes the offending -entry, and the next start goes through. +Absent paths pass, so a first start works. The check neither deletes nor +repairs: the operator removes the offending entry, and the next start goes +through. +The drop-in names the `hive-priv` binary that wrote it, so a drop-in never +points at a build without the sub-command. A drop-in written by an older +points at a build without the sub-command. That build's store path has a GC +root, `/nix/var/nix/gcroots/hyperhive-check-rootfs/h-`, written before the +drop-in and replaced on each write, so a garbage collection can't remove the +binary a restart would run. The directory sits beside `gcroots/per-container`, +which is the only gcroots path bound into a container, so only root writes it. +Destroying the container removes the drop-in and then its root; the build stays +until the next collection. A drop-in written by an older hive-priv has no +`ExecStartPre=` and no root, and the container starts as before. + +**Hyperhive's platform containers: warn only.** The forge, the matrix server and +the swarm services (15 containers, declared by their own modules) run the same +check as `hive-priv check-rootfs --warn-only `, from `ExecStartPre=` on +their own `container@` unit. It evaluates every rule, logs each violation +as a warning, and exits 0, so the container starts either way. The line reads +`rootfs check (warn-only): : ` with journald's warning priority. +Strict refusal needs the live roots of all 15 to pass first, and the warnings +show which ones would not. A container the operator runs that hyperhive doesn't +declare gets no check at all: the modules name their own containers, and +nothing enumerates the host's `containers`. **Disk use.** The copy shares blocks with the old root where it can: hive-priv clones each file (`FICLONE`), which on btrfs or xfs allocates metadata only, and diff --git a/docs/trust-boundary/security.md b/docs/trust-boundary/security.md index 7f076c1d..3d7412e0 100644 --- a/docs/trust-boundary/security.md +++ b/docs/trust-boundary/security.md @@ -263,8 +263,8 @@ known operations; there is no arbitrary command pass-through: | `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) | | `WriteNspawnFlags` | write `/etc/nixos-containers/.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted | | `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else | -| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container | -| `RemoveServiceDropin` | remove `container@.service.d/` drop-in on destroy | +| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs ` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names | +| `RemoveServiceDropin` | remove `container@.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named | | `DaemonReload` | `systemctl daemon-reload` | | `ControlInfraContainer` | `systemctl nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) | | `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` | diff --git a/hive-priv-sock/src/lib.rs b/hive-priv-sock/src/lib.rs index d8e066fd..623fe470 100644 --- a/hive-priv-sock/src/lib.rs +++ b/hive-priv-sock/src/lib.rs @@ -369,7 +369,11 @@ pub enum PrivRequest { /// Write `/run/systemd/system/container@.service.d/hyperhive-limits.conf` /// with `[Service]` carrying `MemoryMax=` / `CPUQuota=` (hard caps) and - /// `CPUWeight=` / `IOWeight=` (cgroup v2 relative shares, contention-only). + /// `CPUWeight=` / `IOWeight=` (cgroup v2 relative shares, contention-only), + /// and an `ExecStartPre=` that runs `hive-priv check-rootfs ` + /// to refuse a start over a root with links where the start script + /// expects directories. The build that line names is rooted under + /// `/nix/var/nix/gcroots/hyperhive-check-rootfs/`. /// Written by `lifecycle::set_resource_limits`. WriteResourceLimits { container: String, @@ -388,7 +392,8 @@ pub enum PrivRequest { io_weight: Option, }, - /// Remove `/run/systemd/system/container@.service.d/` if present. + /// Remove `/run/systemd/system/container@.service.d/` if present, + /// and the GC root of the hive-priv build its `ExecStartPre=` named. /// Called by `lifecycle::destroy` to clean up the resource-limits drop-in. RemoveServiceDropin { container: String }, diff --git a/hive-priv/src/container_config.rs b/hive-priv/src/container_config.rs index e5371e97..197747c7 100644 --- a/hive-priv/src/container_config.rs +++ b/hive-priv/src/container_config.rs @@ -91,7 +91,7 @@ pub(crate) async fn handle_write_nspawn_flags_blocking( /// the manager included) and return the logical agent name after it. /// Anything else is refused, the swarm's forge and matrix containers /// included: they belong to the host that runs them, not to a hive. -fn validate_container_system_name(name: &str) -> Result<&str> { +pub(crate) fn validate_container_system_name(name: &str) -> Result<&str> { let Some(suffix) = name.strip_prefix(AGENT_PREFIX) else { bail!("container name {name:?} is not managed by hive"); }; @@ -100,13 +100,14 @@ fn validate_container_system_name(name: &str) -> Result<&str> { } /// `RemoveServiceDropin` — remove the container service's drop-in dir -/// if present (idempotent). +/// if present (idempotent), then the GC root of the build it named. pub(crate) fn remove_service_dropin(container: &str) -> Result<(String, String)> { validate_container_system_name(container)?; let dir = format!("/run/systemd/system/container@{container}.service.d"); if Path::new(&dir).exists() { std::fs::remove_dir_all(&dir).with_context(|| format!("remove {dir}"))?; } + crate::prestart::unpin_binary(container)?; Ok((String::new(), String::new())) } @@ -114,6 +115,13 @@ pub(crate) fn remove_service_dropin(container: &str) -> Result<(String, String)> /// container service's drop-in dir, together with a /// `ConditionPathIsDirectory=` guard on the agent's MCP runtime dir. /// +/// The drop-in also carries the `ExecStartPre=` that runs +/// `hive-priv check-rootfs ` ([`crate::prestart`]) ahead of +/// nixpkgs' start script. It names this process's own binary, so the drop-in +/// and the binary that implements the check are always the same build. The +/// build is rooted against garbage collection for as long as the drop-in +/// names it ([`crate::prestart::pin_binary`]). +/// /// Two different kinds of setting land in the same file. `MemoryMax=` / /// `CPUQuota=` are hard caps that throttle even on an idle host; /// `CPUWeight=` / `IOWeight=` are cgroup v2 relative shares that only @@ -139,7 +147,19 @@ pub(crate) fn write_resource_limits( let dir = format!("/run/systemd/system/container@{container}.service.d"); std::fs::create_dir_all(&dir).with_context(|| format!("create {dir}"))?; let path = format!("{dir}/hyperhive-limits.conf"); - let content = limits_dropin_body(&runtime_dir, memory_max, cpu_quota, cpu_weight, io_weight); + let exe = hive_priv_exe()?; + // Rooted before the drop-in names it, so the drop-in never names a build a + // collection can take. + crate::prestart::pin_binary(container, Path::new(&exe))?; + let rootfs_check = rootfs_check_command(&exe, container); + let content = limits_dropin_body( + &runtime_dir, + &rootfs_check, + memory_max, + cpu_quota, + cpu_weight, + io_weight, + ); // Published whole: a `daemon-reload` between truncate and write would // otherwise load the unit with its limits and start guard missing. publish_file(Path::new(&path), content.as_bytes(), 0o644, None)?; @@ -175,6 +195,10 @@ const RESTART_SEC: u32 = 5; /// Render the body of `hyperhive-limits.conf`. /// +/// `rootfs_check` is the command line of the pre-start root check, the first +/// `[Service]` line: it runs before the nixpkgs start script and a non-zero +/// exit fails the start. +/// /// `[Unit]`: the condition is checked at start time — it skips (not fails) /// the unit when the MCP socket dir is absent, avoiding restart loops — /// plus the bounded start limit (see the constants above; `StartLimit*` are @@ -186,6 +210,7 @@ const RESTART_SEC: u32 = 5; /// existed — or one whose nix option is `null` — renders no weight lines. fn limits_dropin_body( runtime_dir: &str, + rootfs_check: &str, memory_max: &str, cpu_quota: &str, cpu_weight: Option, @@ -203,6 +228,7 @@ fn limits_dropin_body( StartLimitBurst={START_LIMIT_BURST}\n\ \n\ [Service]\n\ + ExecStartPre={rootfs_check}\n\ RestartSec={RESTART_SEC}\n\ MemoryMax={memory_max}\n\ CPUQuota={cpu_quota}\n\ @@ -210,6 +236,36 @@ fn limits_dropin_body( ) } +/// This binary's absolute path, for the drop-in's `ExecStartPre=`. +/// +/// The path is the running binary's, not a fixed one, so a drop-in never names +/// a build without the sub-command. systemd expands `%` and `$` in command +/// lines and splits on spaces, so a path with any of them is refused rather +/// than escaped; store paths never have them. +fn hive_priv_exe() -> Result { + let exe = std::env::current_exe().context("resolve the hive-priv binary path")?; + let exe = exe + .to_str() + .filter(|p| { + p.starts_with('/') + && p.bytes() + .all(|b| b.is_ascii_alphanumeric() || b"/._+-".contains(&b)) + }) + .with_context(|| { + format!( + "hive-priv binary path {} is not safe in a unit file", + exe.display() + ) + })?; + Ok(exe.to_owned()) +} + +/// The `ExecStartPre=` command of the pre-start root check for `container`: +/// the binary `exe`, then the sub-command and the name. +fn rootfs_check_command(exe: &str, container: &str) -> String { + format!("{exe} {} {container}", crate::prestart::SUBCOMMAND) +} + /// `DaemonReload` — `systemctl daemon-reload` on the host. pub(crate) async fn daemon_reload() -> Result<(String, String)> { let out = Command::new("systemctl") @@ -568,11 +624,15 @@ pub(crate) fn write_container_etc_file_in(rootfs: &Path, name: &str, contents: & mod tests { use super::{ BindMount, HiveFacts, NetworkIsolation, git_overlay_flags, handle_write_nspawn_flags, - limits_dropin_body, nspawn_flags, publish_file, remove_service_dropin, render_hive_facts, - validate_container_system_name, write_bridge_dns_marker_in, write_resource_limits, + hive_priv_exe, limits_dropin_body, nspawn_flags, publish_file, remove_service_dropin, + render_hive_facts, rootfs_check_command, validate_container_system_name, + write_bridge_dns_marker_in, write_resource_limits, }; use crate::shared::test_support::{entries, scratch}; + /// A stand-in for the pre-start check's command line. + const CHECK: &str = "/store/hive-priv/bin/hive-priv check-rootfs h-iris"; + fn bind(container_path: &str) -> BindMount { BindMount { host_path: "/var/lib/hyperhive/whatever".to_owned(), @@ -630,13 +690,21 @@ mod tests { #[test] fn dropin_body_is_pinned_exactly() { assert_eq!( - limits_dropin_body("/run/hyperhive/agents/iris", "4G", "200%", None, None), + limits_dropin_body( + "/run/hyperhive/agents/iris", + CHECK, + "4G", + "200%", + None, + None + ), "[Unit]\n\ ConditionPathIsDirectory=/run/hyperhive/agents/iris\n\ StartLimitIntervalSec=600\n\ StartLimitBurst=3\n\ \n\ [Service]\n\ + ExecStartPre=/store/hive-priv/bin/hive-priv check-rootfs h-iris\n\ RestartSec=5\n\ MemoryMax=4G\n\ CPUQuota=200%\n" @@ -654,7 +722,7 @@ mod tests { ("512M", "50%", Some(10), Some(10)), ("infinity", "infinity", Some(10_000), None), ] { - let body = limits_dropin_body("/rt/x", mem, cpu, cw, iw); + let body = limits_dropin_body("/rt/x", CHECK, mem, cpu, cw, iw); let unit = body .split("[Service]") .next() @@ -674,26 +742,55 @@ mod tests { /// independently when `None`. #[test] fn weights_are_emitted_only_when_set() { - let both = limits_dropin_body("/rt/x", "4G", "200%", Some(80), Some(80)); + let both = limits_dropin_body("/rt/x", CHECK, "4G", "200%", Some(80), Some(80)); assert!( both.ends_with("CPUQuota=200%\nCPUWeight=80\nIOWeight=80\n"), "{both}" ); - let cpu_only = limits_dropin_body("/rt/x", "4G", "200%", Some(80), None); + let cpu_only = limits_dropin_body("/rt/x", CHECK, "4G", "200%", Some(80), None); assert!( cpu_only.ends_with("CPUQuota=200%\nCPUWeight=80\n"), "{cpu_only}" ); assert!(!cpu_only.contains("IOWeight"), "{cpu_only}"); - let io_only = limits_dropin_body("/rt/x", "4G", "200%", None, Some(80)); + let io_only = limits_dropin_body("/rt/x", CHECK, "4G", "200%", None, Some(80)); assert!( io_only.ends_with("CPUQuota=200%\nIOWeight=80\n"), "{io_only}" ); assert!(!io_only.contains("CPUWeight"), "{io_only}"); } + + /// The pre-start check is the first `[Service]` line, whatever else the + /// caller sets, and a drop-in without it is what an old hive-priv wrote. + #[test] + fn the_rootfs_check_leads_the_service_section_whatever_the_caller_passes() { + for (cw, iw) in [(None, None), (Some(80), Some(80))] { + let body = limits_dropin_body("/rt/x", CHECK, "4G", "200%", cw, iw); + let service = body + .split("[Service]\n") + .nth(1) + .expect("the body always has a [Service] section"); + assert_eq!( + service.lines().next(), + Some("ExecStartPre=/store/hive-priv/bin/hive-priv check-rootfs h-iris") + ); + assert_eq!(body.matches("ExecStartPre=").count(), 1, "{body}"); + } + } + + /// The command names an absolute path to this very binary, the + /// sub-command that binary implements and the container. + #[test] + fn the_check_command_names_this_binary_the_subcommand_and_the_container() { + let exe = std::env::current_exe().unwrap(); + let cmd = rootfs_check_command(&hive_priv_exe().unwrap(), "h-iris"); + assert_eq!(cmd, format!("{} check-rootfs h-iris", exe.display())); + assert!(cmd.starts_with('/'), "{cmd}"); + } + /// The mode lands exactly as asked, whatever the umask would have made /// of it (0666 under the usual 022 would otherwise come out 0644), and /// the requested owner is on the file. diff --git a/hive-priv/src/prestart.rs b/hive-priv/src/prestart.rs index 5bfbfb8f..b0a0683e 100644 --- a/hive-priv/src/prestart.rs +++ b/hive-priv/src/prestart.rs @@ -3,30 +3,44 @@ //! nixpkgs' `container@.service` start script runs as host root against the //! container's own root, which is persistent and writable by the container. //! `mkdir -p`, `chmod`, `touch` and `cp --remove-destination` there follow a -//! link the container planted at any of the names the script touches. A -//! drop-in of the `container@` template -//! (`nix/host-modules/container-rootfs-check.nix`) runs -//! `hive-priv check-rootfs %i` as an `ExecStartPre=` ahead of that script for -//! every instance; a non-zero exit fails the start. The container is stopped +//! link the container planted at any of the names the script touches. An agent +//! container's drop-in ([`crate::container_config`]) puts +//! `hive-priv check-rootfs ` ahead of that script as an +//! `ExecStartPre=`; a non-zero exit fails the start. The container is stopped //! while this runs, so nothing changes under the walk. //! +//! Hyperhive's own declarative containers (the forge, the matrix server, the +//! swarm services) run it as `check-rootfs --warn-only ` from their +//! own unit (`nix/host-modules/lib/container-rootfs-check.nix`): each violation +//! is logged at warning level and the start goes on. +//! //! Every component is looked at with `symlink_metadata`: a link is reported, //! never followed. Nothing is deleted or repaired, so whatever is there stays //! for the operator. //! +//! The drop-in names the running hive-priv binary, which nothing else roots +//! once the system has moved on to a newer build. [`pin_binary`] keeps a GC +//! root on it for as long as the drop-in exists, and [`unpin_binary`] drops +//! the root with the drop-in. +//! //! The entry point is [`run`]; [`check`] holds the rules. use std::fs::{FileType, Metadata}; +use std::os::unix::fs::{DirBuilderExt as _, symlink}; use std::path::{Path, PathBuf}; use anyhow::{Context as _, Result, bail}; -use crate::container_config::container_rootfs; -use crate::shared::validate_name_chars; +use crate::container_config::{container_rootfs, validate_container_system_name}; +use crate::shared::{partial_name, validate_name_chars}; -/// The sub-command the template drop-in's `ExecStartPre=` names. +/// The sub-command the drop-in's `ExecStartPre=` names. pub(crate) const SUBCOMMAND: &str = "check-rootfs"; +/// The flag that turns a violation into a warning: the check runs in full and +/// exits 0. +const WARN_ONLY: &str = "--warn-only"; + /// The per-container nix profile directories on the host. const PROFILES_ROOT: &str = "/nix/var/nix/profiles/per-container"; @@ -59,47 +73,157 @@ const REGULAR_ONLY: [&str; 2] = ["etc/machine-id", "etc/resolv.conf"]; /// `/etc/static` once it has booted. const OS_RELEASE: &str = "etc/os-release"; +/// Where hive-priv roots the build named by each container's `ExecStartPre=`: +/// `` is a symlink to that build's store path. The directory is +/// under the host's `/nix/var/nix/gcroots`, not `per-container`, which is the +/// only part of it a container is given (`--bind=.../gcroots/per-container/` +/// in the nixpkgs start script), so a container can't reach it. +const GCROOTS: &str = "/nix/var/nix/gcroots/hyperhive-check-rootfs"; + +/// Root the store path of `exe` for `container`, replacing the root a previous +/// write left, so the build a drop-in names outlives a garbage collection for +/// as long as the drop-in does. One link per container, so they don't pile up. +pub(crate) fn pin_binary(container: &str, exe: &Path) -> Result<()> { + pin_binary_in(Path::new(GCROOTS), Path::new(STORE), container, exe) +} + +/// Drop the root [`pin_binary`] left for `container`, once its drop-in is gone. +/// A name that holds anything but a link is left alone and reported. +pub(crate) fn unpin_binary(container: &str) -> Result<()> { + unpin_binary_in(Path::new(GCROOTS), container) +} + +/// [`pin_binary`] with the roots directory and the store as parameters. +/// +/// A binary outside `store` can't be collected, so it gets no root. The link +/// is made under a temporary name and renamed over ``: `rename` +/// replaces whatever link is there without following it. +fn pin_binary_in(roots: &Path, store: &Path, container: &str, exe: &Path) -> Result<()> { + let Some(entry) = exe + .strip_prefix(store) + .ok() + .and_then(|rest| rest.components().next()) + else { + return Ok(()); + }; + let target = store.join(entry); + std::fs::DirBuilder::new() + .recursive(true) + .mode(0o755) + .create(roots) + .with_context(|| format!("create {}", roots.display()))?; + if !lstat(roots)?.is_some_and(|m| m.file_type().is_dir()) { + bail!("{} is not a real directory", roots.display()); + } + let tmp = roots.join(partial_name(container)); + symlink(&target, &tmp).with_context(|| format!("create {}", tmp.display()))?; + let root = roots.join(container); + std::fs::rename(&tmp, &root).map_err(|e| { + // Best effort: a leftover temp is a root too, but a harmless one. + let _ = std::fs::remove_file(&tmp); + anyhow::Error::new(e).context(format!("replace {}", root.display())) + }) +} + +/// [`unpin_binary`] with the roots directory as a parameter. +fn unpin_binary_in(roots: &Path, container: &str) -> Result<()> { + let root = roots.join(container); + match lstat(&root)? { + None => Ok(()), + Some(meta) if meta.file_type().is_symlink() => { + std::fs::remove_file(&root).with_context(|| format!("remove {}", root.display())) + } + Some(meta) => bail!( + "{} is {}, not a link: left alone", + root.display(), + kind(meta.file_type()) + ), + } +} + /// Handle `hive-priv ...` when the first argument names one. /// `None` means it is not a sub-command and the daemon starts. /// -/// Prints one line to stderr and returns exit code 1 on a violation, so the -/// reason lands in the journal next to the failed unit. +/// `check-rootfs ` is strict: it prints one line to stderr and +/// returns exit code 1 on a violation, so the reason lands in the journal next +/// to the failed unit. `check-rootfs --warn-only ` prints every +/// violation as a warning and returns 0. pub(crate) fn run(args: &[String]) -> Option { if args.first().map(String::as_str) != Some(SUBCOMMAND) { return None; } - let result = match args { - [_, container] => check_container(container), - _ => Err(anyhow::anyhow!("usage: hive-priv {SUBCOMMAND} ")), - }; - Some(match result { - Ok(()) => 0, - Err(e) => { - eprintln!("hive-priv {SUBCOMMAND}: refusing to start: {e:#}"); - 1 + let (container, warn_only) = match args { + [_, container] => (container, false), + [_, flag, container] if flag == WARN_ONLY => (container, true), + _ => { + eprintln!("usage: hive-priv {SUBCOMMAND} [{WARN_ONLY}] "); + return Some(1); } - }) + }; + let found = found_in_container(container, warn_only); + let (code, lines) = report(container, &found, warn_only); + for line in lines { + eprintln!("{line}"); + } + Some(code) } -/// [`check`] on the host paths of `container`. -fn check_container(container: &str) -> Result<()> { - let (root, system) = container_paths(container)?; - check(&root, &system, Path::new(STORE)) +/// What a start would be refused for, on the host paths of `container`. +/// +/// Strict mode stops at the first violation, as the start would. Warn-only +/// lists them all, and counts a name it can't use as one more: it never fails +/// the start over its own input. +fn found_in_container(container: &str, warn_only: bool) -> Vec { + let paths = container_paths(container, warn_only); + match (paths, warn_only) { + (Ok((root, system)), true) => violations(&root, &system, Path::new(STORE)), + (Ok((root, system)), false) => check(&root, &system, Path::new(STORE)) + .err() + .map(|e| format!("{e:#}")) + .into_iter() + .collect(), + (Err(e), _) => vec![format!("{e:#}")], + } } /// The container's root and its profile's `system` link on the host. /// -/// The unit instance supplies the name and it selects both paths, so anything -/// but a plain container name is refused: lowercase ascii, digits and -/// hyphens. That holds no `/` and is never `.` or `..`. -fn container_paths(container: &str) -> Result<(PathBuf, PathBuf)> { - validate_name_chars(container)?; +/// An agent container is named `h-`. A warn-only check also takes the +/// platform containers' names, which are lowercase ascii, digits and hyphens. +/// Either way the name holds no `/` and is never `.` or `..`. +fn container_paths(container: &str, warn_only: bool) -> Result<(PathBuf, PathBuf)> { + if warn_only { + validate_name_chars(container)?; + } else { + validate_container_system_name(container)?; + } Ok(( container_rootfs(container), Path::new(PROFILES_ROOT).join(container).join("system"), )) } +/// The exit code and the stderr lines for the violations `found`. +/// +/// Warn-only lines carry the `<4>` prefix, which journald reads as the +/// warning priority, so a log query can select them by severity. +fn report(container: &str, found: &[String], warn_only: bool) -> (i32, Vec) { + if warn_only { + let lines = found + .iter() + .map(|v| format!("<4>rootfs check (warn-only): {container}: {v}")) + .collect(); + return (0, lines); + } + match found.first() { + Some(v) => ( + 1, + vec![format!("hive-priv {SUBCOMMAND}: refusing to start: {v}")], + ), + None => (0, Vec::new()), + } +} + /// Check the paths the start script touches as root, returning an error that /// names the first path to hold something it must not. /// @@ -109,14 +233,36 @@ fn container_paths(container: &str) -> Result<(PathBuf, PathBuf)> { /// A path that does not exist passes: a first start has nothing yet, and the /// script creates it. fn check(root: &Path, system: &Path, store: &Path) -> Result<()> { + match violations(root, system, store).into_iter().next() { + None => Ok(()), + Some(first) => bail!("{first}"), + } +} + +/// Every path under the rules of [`check`] that holds something it must not, +/// in the order [`check`] looks at them, each message once. +fn violations(root: &Path, system: &Path, store: &Path) -> Vec { + let mut found = Vec::new(); for dir in DIRS { - check_dir(root, dir)?; + note(&mut found, check_dir(root, dir)); } for file in REGULAR_ONLY { - check_file(root, file, false)?; + note(&mut found, check_file(root, file, false)); + } + note(&mut found, check_file(root, OS_RELEASE, true)); + note(&mut found, check_system(system, store)); + found +} + +/// Add the message of an error to `found`, unless it is already there: a link +/// at `usr` is the finding for `usr/bin` too. +fn note(found: &mut Vec, result: Result<()>) { + if let Err(e) = result { + let message = format!("{e:#}"); + if !found.contains(&message) { + found.push(message); + } } - check_file(root, OS_RELEASE, true)?; - check_system(system, store) } /// `symlink_metadata`, with `None` for a path that does not exist. The one @@ -221,8 +367,8 @@ mod tests { use std::os::unix::fs::symlink; use std::path::PathBuf; - use super::{check, container_paths, run}; - use crate::shared::test_support::scratch; + use super::{check, container_paths, pin_binary_in, report, run, unpin_binary_in, violations}; + use crate::shared::test_support::{entries, scratch}; /// A scratch container: root, the profile link's path and a store with one /// toplevel in it. Nothing is linked yet. @@ -450,11 +596,133 @@ mod tests { assert!(target.is_dir()); } + /// A scratch store holding one build with a binary in it, a roots dir and + /// the binary's path. + fn pin_fixture() -> (PathBuf, PathBuf, PathBuf, PathBuf) { + let base = scratch(); + let store = base.join("store"); + let build = store.join("abc-hive-priv"); + std::fs::create_dir_all(build.join("bin")).unwrap(); + let exe = build.join("bin/hive-priv"); + std::fs::write(&exe, "").unwrap(); + (base.join("roots"), store, build, exe) + } + #[test] - fn a_plain_container_name_selects_its_root_and_profile() { - // Agent and non-agent names alike. - for name in ["h-iris", "hive-forge", "swarm-review-classifier"] { - let (root, system) = container_paths(name).unwrap(); + fn the_root_points_at_the_store_path_the_drop_in_names() { + let (roots, store, build, exe) = pin_fixture(); + pin_binary_in(&roots, &store, "h-iris", &exe).unwrap(); + let root = roots.join("h-iris"); + assert!(std::fs::symlink_metadata(&root).unwrap().is_symlink()); + // The build, not the binary inside it. + assert_eq!(std::fs::read_link(&root).unwrap(), build); + assert_eq!(std::fs::read_dir(&roots).unwrap().count(), 1); + } + + #[test] + fn a_rewrite_replaces_the_root_and_leaves_no_temp() { + let (roots, store, build, exe) = pin_fixture(); + let newer = store.join("def-hive-priv"); + std::fs::create_dir_all(newer.join("bin")).unwrap(); + pin_binary_in(&roots, &store, "h-iris", &exe).unwrap(); + pin_binary_in(&roots, &store, "h-iris", &newer.join("bin/hive-priv")).unwrap(); + pin_binary_in(&roots, &store, "h-bob", &exe).unwrap(); + assert_eq!(std::fs::read_link(roots.join("h-iris")).unwrap(), newer); + assert_eq!(std::fs::read_link(roots.join("h-bob")).unwrap(), build); + assert_eq!(entries(&roots), ["h-bob", "h-iris"]); + } + + #[test] + fn a_link_planted_at_the_root_name_is_replaced_not_followed() { + let (roots, store, build, exe) = pin_fixture(); + let victim = victim(); + std::fs::write(victim.join("keep"), "data").unwrap(); + std::fs::create_dir_all(&roots).unwrap(); + symlink(&victim, roots.join("h-iris")).unwrap(); + pin_binary_in(&roots, &store, "h-iris", &exe).unwrap(); + assert_eq!(std::fs::read_link(roots.join("h-iris")).unwrap(), build); + assert_eq!( + std::fs::read_to_string(victim.join("keep")).unwrap(), + "data" + ); + assert_eq!(std::fs::read_dir(&victim).unwrap().count(), 1); + } + + #[test] + fn a_roots_dir_that_is_a_link_is_refused() { + let (roots, store, _, exe) = pin_fixture(); + let victim = victim(); + symlink(&victim, &roots).unwrap(); + let msg = format!( + "{:#}", + pin_binary_in(&roots, &store, "h-iris", &exe).unwrap_err() + ); + assert!(msg.contains("is not a real directory"), "{msg}"); + assert_eq!(std::fs::read_dir(&victim).unwrap().count(), 0); + } + + #[test] + fn a_binary_outside_the_store_gets_no_root() { + let (roots, store, _, _) = pin_fixture(); + let elsewhere = scratch().join("hive-priv"); + pin_binary_in(&roots, &store, "h-iris", &elsewhere).unwrap(); + assert!(!roots.exists()); + } + + #[test] + fn unpinning_removes_the_container_root_and_nothing_else() { + let (roots, store, build, exe) = pin_fixture(); + pin_binary_in(&roots, &store, "h-iris", &exe).unwrap(); + pin_binary_in(&roots, &store, "h-bob", &exe).unwrap(); + unpin_binary_in(&roots, "h-iris").unwrap(); + assert_eq!(entries(&roots), ["h-bob"]); + // Already gone: fine. The build it named is untouched. + unpin_binary_in(&roots, "h-iris").unwrap(); + assert!(build.join("bin/hive-priv").is_file()); + } + + #[test] + fn unpinning_leaves_a_name_that_is_not_a_link_alone() { + let (roots, _, _, _) = pin_fixture(); + std::fs::create_dir_all(roots.join("h-iris")).unwrap(); + std::fs::write(roots.join("h-iris/keep"), "").unwrap(); + let msg = format!("{:#}", unpin_binary_in(&roots, "h-iris").unwrap_err()); + assert!(msg.contains("left alone"), "{msg}"); + assert!(roots.join("h-iris/keep").exists()); + } + + #[test] + fn only_the_check_rootfs_sub_command_is_taken() { + let args = |a: &[&str]| a.iter().map(|s| (*s).to_owned()).collect::>(); + assert_eq!(run(&[]), None); + assert_eq!(run(&args(&["serve"])), None); + // Wrong arity, an unknown flag and a name that is not an agent + // container are refused, whatever is on disk. + assert_eq!(run(&args(&["check-rootfs"])), Some(1)); + assert_eq!(run(&args(&["check-rootfs", "not-hive"])), Some(1)); + assert_eq!(run(&args(&["check-rootfs", "--strict", "h-iris"])), Some(1)); + assert_eq!(run(&args(&["check-rootfs", "--warn-only"])), Some(1)); + } + + #[test] + fn a_warn_only_run_exits_zero_even_for_a_name_it_cannot_use() { + let args = |a: &[&str]| a.iter().map(|s| (*s).to_owned()).collect::>(); + assert_eq!( + run(&args(&["check-rootfs", "--warn-only", "no-such-container"])), + Some(0) + ); + assert_eq!( + run(&args(&["check-rootfs", "--warn-only", "../etc"])), + Some(0) + ); + } + + #[test] + fn strict_takes_agent_names_and_warn_only_takes_plain_ones() { + assert!(container_paths("h-iris", false).is_ok()); + assert!(container_paths("hive-forge", false).is_err()); + for name in ["hive-forge", "swarm-review-classifier", "h-iris"] { + let (root, system) = container_paths(name, true).unwrap(); assert_eq!( root, PathBuf::from(format!("/var/lib/nixos-containers/{name}")) @@ -464,10 +732,6 @@ mod tests { PathBuf::from(format!("/nix/var/nix/profiles/per-container/{name}/system")) ); } - } - - #[test] - fn a_name_that_is_not_a_plain_container_name_is_refused() { for name in [ "", ".", @@ -475,28 +739,84 @@ mod tests { "../h-iris", "a/b", "/etc", - "h-iris/", "H-iris", "a.b", "a b", ] { - assert!(container_paths(name).is_err(), "{name:?} was accepted"); - } - } - - #[test] - fn only_the_check_rootfs_sub_command_is_taken() { - assert_eq!(run(&[]), None); - assert_eq!(run(&["serve".to_string()]), None); - // Wrong arity and a name that selects a path outside the containers - // directory are refused, whatever is on disk. - assert_eq!(run(&["check-rootfs".to_string()]), Some(1)); - for name in ["", "..", "a/b"] { - assert_eq!( - run(&["check-rootfs".to_string(), name.to_string()]), - Some(1), - "{name:?}" + assert!( + container_paths(name, true).is_err(), + "{name:?} was accepted" + ); + assert!( + container_paths(name, false).is_err(), + "{name:?} was accepted" ); } } + + /// A clean root with a link at `root` and another at `etc/resolv.conf`. + fn two_violations() -> Fixture { + let f = Fixture::clean(); + std::fs::remove_dir_all(f.root.join("root")).unwrap(); + symlink(victim(), f.root.join("root")).unwrap(); + std::fs::remove_file(f.root.join("etc/resolv.conf")).unwrap(); + symlink("/etc/hosts", f.root.join("etc/resolv.conf")).unwrap(); + f + } + + fn found(f: &Fixture) -> Vec { + violations(&f.root, &f.system, &f.store) + } + + #[test] + fn warn_only_logs_every_violation_at_warning_level_and_exits_zero() { + let f = two_violations(); + let found = found(&f); + assert_eq!(found.len(), 2, "{found:?}"); + let (code, lines) = report("hive-forge", &found, true); + assert_eq!(code, 0); + assert_eq!(lines.len(), 2, "{lines:?}"); + for line in &lines { + assert!( + line.starts_with("<4>rootfs check (warn-only): hive-forge: "), + "{line}" + ); + } + assert!(lines[0].contains("root is a symlink"), "{lines:?}"); + assert!(lines[1].contains("resolv.conf is a symlink"), "{lines:?}"); + } + + #[test] + fn strict_refuses_with_the_first_violation_only() { + let f = two_violations(); + let found = found(&f); + let (code, lines) = report("h-iris", &found, false); + assert_eq!(code, 1); + assert_eq!(lines.len(), 1, "{lines:?}"); + assert!( + lines[0].starts_with("hive-priv check-rootfs: refusing to start: "), + "{lines:?}" + ); + assert!(lines[0].contains("root is a symlink"), "{lines:?}"); + // The rules are the same: the strict refusal is `check`'s first error. + assert_eq!(f.refused(), found[0]); + } + + #[test] + fn a_clean_root_prints_nothing_in_either_mode() { + let f = Fixture::clean(); + assert!(found(&f).is_empty()); + assert_eq!(report("hive-forge", &[], true), (0, Vec::new())); + assert_eq!(report("h-iris", &[], false), (0, Vec::new())); + } + + #[test] + fn one_link_is_reported_once_though_two_paths_run_through_it() { + let f = Fixture::clean(); + std::fs::remove_dir_all(f.root.join("usr")).unwrap(); + symlink(victim(), f.root.join("usr")).unwrap(); + let found = found(&f); + assert_eq!(found.len(), 1, "{found:?}"); + assert!(found[0].contains("usr is a symlink"), "{found:?}"); + } } diff --git a/hive-priv/src/shared.rs b/hive-priv/src/shared.rs index 0c06f35e..a8ae7332 100644 --- a/hive-priv/src/shared.rs +++ b/hive-priv/src/shared.rs @@ -75,7 +75,7 @@ pub(crate) fn ensure_plain_filename(who: &str, filename: &str) -> Result<()> { /// The leading dot and the `.partial` suffix are load-bearing: a reader /// globbing on a published name's prefix, or systemd reading `*.conf` out of /// `tmpfiles.d` and drop-in dirs, cannot pick up a half-written temp. -fn partial_name(filename: &str) -> String { +pub(crate) fn partial_name(filename: &str) -> String { use std::sync::atomic::{AtomicU64, Ordering}; static SEQ: AtomicU64 = AtomicU64::new(0); let n = SEQ.fetch_add(1, Ordering::Relaxed); diff --git a/nix/host-modules/container-rootfs-check.nix b/nix/host-modules/container-rootfs-check.nix deleted file mode 100644 index 3006df81..00000000 --- a/nix/host-modules/container-rootfs-check.nix +++ /dev/null @@ -1,44 +0,0 @@ -# The pre-start check of a container's root, for every `container@` instance. -# -# nixpkgs' start script runs as host root against the container's own root, -# which is persistent and writable by the container, and its `mkdir`, `chmod`, -# `touch` and `cp` follow links the container planted there. A drop-in on the -# `container@` template runs `hive-priv check-rootfs %i` as an `ExecStartPre=` -# before the script (`hive-priv/src/prestart.rs` has the rules). It covers the -# agents' imperative instances and the declarative ones (hive-forge, -# hive-matrix, the swarm services), each of which has a unit file of its own: -# systemd reads the template's drop-in directory for every instance. -# -# Order: systemd parses a drop-in after the unit file, so this line follows the -# unit file's own `ExecStartPre=` (nixpkgs' `preStart`, which touches no path -# under the root), and `ExecStart=` runs only after every `ExecStartPre=` -# succeeded. -# -# The drop-in is a unit-file package rather than `systemd.services."container@"` -# because that attribute renders the template's own unit file, which a -# declarative instance's unit file replaces. -{ - config, - lib, - pkgs, - ... -}: -let - dropin = pkgs.writeTextFile { - name = "hyperhive-container-rootfs-check"; - destination = "/lib/systemd/system/container@.service.d/10-hyperhive-rootfs-check.conf"; - text = '' - [Service] - ExecStartPre=${config.services.hyperhive.c0re.package}/bin/hive-priv check-rootfs %i - ''; - }; -in -{ - config = lib.mkIf config.boot.enableContainers { - systemd.packages = [ dropin ]; - - # The per-agent drop-ins this check replaced rooted a hive-priv build per - # agent here. Boot is the one time no such drop-in is loaded. - systemd.tmpfiles.rules = [ "R! /nix/var/nix/gcroots/hyperhive-check-rootfs - - - - -" ]; - }; -} diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index 89f19214..17039af3 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -22,7 +22,6 @@ ./hive-matrix-service.nix ./hive-matrix.nix ./hive-network.nix - ./container-rootfs-check.nix ./hive-priv.nix ./hive-priv-cache.nix ./hive-squid.nix diff --git a/nix/host-modules/hive-forge/default.nix b/nix/host-modules/hive-forge/default.nix index 647ea995..565bbbc3 100644 --- a/nix/host-modules/hive-forge/default.nix +++ b/nix/host-modules/hive-forge/default.nix @@ -78,6 +78,8 @@ let deployCfg.bao.forgePublisherClientCertFile != null && deployCfg.bao.forgePublisherClientKeyFile != null; + rootfsCheck = import ../lib/container-rootfs-check.nix { inherit config; }; + caTrust = import ../lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; }; # The container's journal forwarder pushes to its own receiver on the swarm @@ -471,6 +473,7 @@ in # ../swarm-builder.nix's does. systemd.services."container@hive-forge" = lib.mkMerge [ caTrust.containerOrdering + (rootfsCheck.warnOnly "hive-forge") { after = [ "swarm-bao-pki.service" ]; wants = [ "swarm-bao-pki.service" ]; diff --git a/nix/host-modules/hive-matrix.nix b/nix/host-modules/hive-matrix.nix index d691cbba..c77a847f 100644 --- a/nix/host-modules/hive-matrix.nix +++ b/nix/host-modules/hive-matrix.nix @@ -13,6 +13,8 @@ let # the client is pointed at the server name instead. hyperhiveDomain = config.services.hyperhive.domain; + rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; + # Same runtime→build-time bridge hive-forge and swarm-ci-runner cross: # binds the hive trust bundle (which folds in the store's root) into the # container and orders the container after `hive-tls-ca.service`. The @@ -1096,6 +1098,7 @@ in systemd.services."container@hive-matrix" = lib.mkMerge [ { after = [ "dnsmasq.service" ]; } caTrust.containerOrdering + (rootfsCheck.warnOnly "hive-matrix") ]; }; } diff --git a/nix/host-modules/lib/container-rootfs-check.nix b/nix/host-modules/lib/container-rootfs-check.nix new file mode 100644 index 00000000..d8832d5e --- /dev/null +++ b/nix/host-modules/lib/container-rootfs-check.nix @@ -0,0 +1,32 @@ +# The pre-start check of a container's root, for one of hyperhive's own +# declarative containers. +# +# nixpkgs' start script runs as host root against the container's own root, +# which is persistent and writable by the container, and its `mkdir`, `chmod`, +# `touch` and `cp` follow links the container planted there. `hive-priv +# check-rootfs` looks at the paths the script touches (rules in +# `hive-priv/src/prestart.rs`). With `--warn-only` it logs each violation at +# warning level and lets the start go on; the agent containers run it strict +# from their per-agent drop-in (hive-priv's `WriteResourceLimits`). +# +# Pure function, not a NixOS module. A container's module calls it with its +# own name and folds the result into its unit: +# +# rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; +# # systemd.services."container@${machine}" = lib.mkMerge [ (rootfsCheck.warnOnly machine) … ]; +# +# The name list is the callers', never `config.containers`: that also holds +# containers the operator runs that hyperhive doesn't own, and those get +# nothing. nixpkgs renders each declarative container as its own +# `container@` unit (`nixos-containers.nix`, the `mapAttrsToList` that +# builds `container@${name}`), so a setting on that unit reaches that +# container only. +{ config }: +{ + # Attributes for `systemd.services."container@"`. + warnOnly = name: { + serviceConfig.ExecStartPre = [ + "${config.services.hyperhive.c0re.package}/bin/hive-priv check-rootfs --warn-only ${name}" + ]; + }; +} diff --git a/nix/host-modules/swarm-attic.nix b/nix/host-modules/swarm-attic.nix index de203d2e..9b6f0477 100644 --- a/nix/host-modules/swarm-attic.nix +++ b/nix/host-modules/swarm-attic.nix @@ -114,6 +114,8 @@ let # one word. oneWord = path: builtins.match ".*[[:space:]].*" path == null; + rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; + # The forwarder reaches the swarm collector and authelia by name over TLS; in # self-signed mode their leaves chain to the hive CA. caTrust = import ./lib/hive-ca-trust.nix { @@ -226,6 +228,7 @@ in # CA ordering keeps the trust bundle's bind source there first. systemd.services."container@${machine}" = lib.mkMerge [ caTrust.containerOrdering + (rootfsCheck.warnOnly machine) { after = [ "swarm-bao-pki.service" ]; wants = [ "swarm-bao-pki.service" ]; diff --git a/nix/host-modules/swarm-authelia.nix b/nix/host-modules/swarm-authelia.nix index 2e591177..39783827 100644 --- a/nix/host-modules/swarm-authelia.nix +++ b/nix/host-modules/swarm-authelia.nix @@ -183,6 +183,8 @@ let refreshConsumer = import ./lib/refresh-consumer.nix { }; storeSeededSecret = import ./lib/store-seeded-secret.nix { }; + rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; + caTrust = import ./lib/hive-ca-trust.nix { inherit lib gatewayCfg; tlsCfg = deployCfg.hive-controller.tls; @@ -970,6 +972,7 @@ in # not be up. systemd.services."container@${cfg.machine}" = lib.mkMerge [ caTrust.containerOrdering + (rootfsCheck.warnOnly cfg.machine) (lib.mkIf publishes { after = [ "swarm-bao-pki.service" ]; wants = [ "swarm-bao-pki.service" ]; diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index dd5b1bab..9e6df0ea 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -1689,6 +1689,8 @@ let forwarderAuthName = "oauth2client/${swarmOtelCfg.storeProducerName}"; autheliaCfg = hyperhiveCfg.swarm.authelia; + rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; + # Outbound TLS for that collector: it dials the swarm collector's gateway # name and authelia's token endpoint, both leaves under the store's runtime # pki root, which cannot be baked into a derivation. Same helper, same two @@ -4091,6 +4093,7 @@ in # cannot start is a store that is down. systemd.services."container@${cfg.machine}" = lib.mkMerge [ caTrust.containerOrdering + (rootfsCheck.warnOnly cfg.machine) { after = [ "systemd-tmpfiles-resetup.service" ]; } ]; diff --git a/nix/host-modules/swarm-builder.nix b/nix/host-modules/swarm-builder.nix index 1fb12bd7..70234fca 100644 --- a/nix/host-modules/swarm-builder.nix +++ b/nix/host-modules/swarm-builder.nix @@ -52,6 +52,8 @@ let # containers' forwarders all the same. forwarderTelemetryPort = 8892; + rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; + # The daemon reaches authelia, the store and the queue by name over TLS, and # `git` reaches the forge; in self-signed mode their leaves chain to the hive # CA. @@ -165,6 +167,7 @@ in # the unit does not exist. systemd.services."container@${machine}" = lib.mkMerge [ caTrust.containerOrdering + (rootfsCheck.warnOnly machine) { after = [ "swarm-bao-pki.service" ]; wants = [ "swarm-bao-pki.service" ]; diff --git a/nix/host-modules/swarm-ci-runner.nix b/nix/host-modules/swarm-ci-runner.nix index 690b0348..a67cd9c5 100644 --- a/nix/host-modules/swarm-ci-runner.nix +++ b/nix/host-modules/swarm-ci-runner.nix @@ -43,6 +43,8 @@ let # below compares the two, so both come from here. forgeUrl = "https://${forgeCfg.domain}"; + rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; + # The forge's gateway serves a leaf from the store's `pki` mount in # self-signed mode. # The runner daemon is Go, whose `SSL_CERT_FILE` replaces the system store, @@ -329,6 +331,7 @@ in # minting unit in and waits for it, as ./swarm-builder.nix's does. systemd.services."container@${machine}" = lib.mkMerge [ caTrust.containerOrdering + (rootfsCheck.warnOnly machine) { after = [ "swarm-bao-pki.service" ]; wants = [ "swarm-bao-pki.service" ]; diff --git a/nix/host-modules/swarm-claude-login.nix b/nix/host-modules/swarm-claude-login.nix index 1c75147f..c7d3b782 100644 --- a/nix/host-modules/swarm-claude-login.nix +++ b/nix/host-modules/swarm-claude-login.nix @@ -54,6 +54,8 @@ let # containers' forwarders all the same. forwarderTelemetryPort = 8893; + rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; + # The daemon reaches authelia, the store and the queue by name over TLS, and # in self-signed mode their leaves chain to the hive CA. caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; }; @@ -146,6 +148,7 @@ in # missing, so where this host mints the leaf the container waits for it. systemd.services."container@${machine}" = lib.mkMerge [ caTrust.containerOrdering + (rootfsCheck.warnOnly machine) { after = [ "swarm-bao-pki.service" ]; } ]; diff --git a/nix/host-modules/swarm-grafana.nix b/nix/host-modules/swarm-grafana.nix index a5bbb2f4..49853a4e 100644 --- a/nix/host-modules/swarm-grafana.nix +++ b/nix/host-modules/swarm-grafana.nix @@ -24,6 +24,8 @@ let baoCfg = hyperhiveCfg.swarm.bao; baoDeploy = deployCfg.bao; + rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; + caTrust = import ./lib/hive-ca-trust.nix { inherit lib gatewayCfg; tlsCfg = deployCfg.hive-controller.tls; @@ -686,6 +688,7 @@ in # ./swarm-builder.nix's does. systemd.services."container@${cfg.machine}" = lib.mkMerge [ caTrust.containerOrdering + (rootfsCheck.warnOnly cfg.machine) { after = [ "swarm-bao-pki.service" ]; wants = [ "swarm-bao-pki.service" ]; diff --git a/nix/host-modules/swarm-nats.nix b/nix/host-modules/swarm-nats.nix index 45b857e1..845c838c 100644 --- a/nix/host-modules/swarm-nats.nix +++ b/nix/host-modules/swarm-nats.nix @@ -33,6 +33,8 @@ let tlsCfg = config.services.hyperhive.deploy.hive-controller.tls; gatewayCfg = config.services.hyperhive.gateway; + rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; + caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; }; # The responder introspects authelia over https BY NAME. Its HTTP client is # reqwest/rustls, and `rustls-platform-verifier` resolves roots through @@ -961,7 +963,10 @@ in # Order the container after the host CA generator, so the bind source # exists before nspawn sets the mount up. Without it a late CA fails the # container start outright rather than degrading. - systemd.services."container@${machine}" = caTrust.containerOrdering; + systemd.services."container@${machine}" = lib.mkMerge [ + caTrust.containerOrdering + (rootfsCheck.warnOnly machine) + ]; # Deliver the responder's store identity into the container before it # starts. A copy rather than a `bindMounts` entry: nixos-container refuses diff --git a/nix/host-modules/swarm-notify-router.nix b/nix/host-modules/swarm-notify-router.nix index e3ed7431..06e8d492 100644 --- a/nix/host-modules/swarm-notify-router.nix +++ b/nix/host-modules/swarm-notify-router.nix @@ -44,6 +44,8 @@ let # containers' forwarders all the same. forwarderTelemetryPort = 8902; + rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; + # The daemon reaches authelia, the store and the queue by name over TLS, and # in self-signed mode their leaves chain to the hive CA. caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; }; @@ -134,6 +136,7 @@ in # missing, so where this host mints the leaf the container waits for it. systemd.services."container@${machine}" = lib.mkMerge [ caTrust.containerOrdering + (rootfsCheck.warnOnly machine) { after = [ "swarm-bao-pki.service" ]; } ]; diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index 9c73bb81..5504ca60 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -173,6 +173,8 @@ let # that ever names the socket. hiveSockets = lib.genAttrs (lib.attrNames hyperhiveCfg.swarm.hives) socketOf; + rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; + # The swarm's authelia is reached by its gateway name, whose leaf is # issued by the swarm services sub-CA — so this container needs the same # runtime CA trust every other consumer of a swarm-service name needs. @@ -706,7 +708,10 @@ in # The CA bind source is written at runtime by a host unit, so the # container has to start after it — otherwise nspawn sets up a mount # over a file that does not exist yet. - systemd.services."container@${cfg.machine}" = caTrust.containerOrdering; + systemd.services."container@${cfg.machine}" = lib.mkMerge [ + caTrust.containerOrdering + (rootfsCheck.warnOnly cfg.machine) + ]; # A collector that is up but not listening on a socket the gateway names # answers every producer with a 502 and fails no unit. This one fails when diff --git a/nix/host-modules/swarm-review-classifier.nix b/nix/host-modules/swarm-review-classifier.nix index f6fd2d4a..02a110a1 100644 --- a/nix/host-modules/swarm-review-classifier.nix +++ b/nix/host-modules/swarm-review-classifier.nix @@ -57,6 +57,8 @@ let # The standard OTLP/HTTP port, on the container's loopback. forwarderOtlpPort = 4318; + rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; + # The daemon reaches authelia, the store and the queue by name over TLS, and # in self-signed mode their leaves chain to the hive CA. caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; }; @@ -172,6 +174,7 @@ in # the unit does not exist. systemd.services."container@${machine}" = lib.mkMerge [ caTrust.containerOrdering + (rootfsCheck.warnOnly machine) { after = [ "swarm-bao-pki.service" ]; wants = [ "swarm-bao-pki.service" ]; diff --git a/nix/host-modules/swarm-victorialogs.nix b/nix/host-modules/swarm-victorialogs.nix index 559fc1ef..007e8a9d 100644 --- a/nix/host-modules/swarm-victorialogs.nix +++ b/nix/host-modules/swarm-victorialogs.nix @@ -99,6 +99,8 @@ let # one word. oneWord = path: builtins.match ".*[[:space:]].*" path == null; + rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; + # The forwarder reaches the swarm collector and authelia by name over TLS; in # self-signed mode their leaves chain to the hive CA. caTrust = import ./lib/hive-ca-trust.nix { @@ -290,6 +292,7 @@ in # ordering keeps the trust bundle's bind source there first. systemd.services."container@${cfg.machine}" = lib.mkMerge [ caTrust.containerOrdering + (rootfsCheck.warnOnly cfg.machine) { after = [ "swarm-bao-pki.service" ]; wants = [ "swarm-bao-pki.service" ]; diff --git a/nix/host-modules/swarm-victoriametrics.nix b/nix/host-modules/swarm-victoriametrics.nix index 48730bb1..04c849a3 100644 --- a/nix/host-modules/swarm-victoriametrics.nix +++ b/nix/host-modules/swarm-victoriametrics.nix @@ -198,6 +198,7 @@ let storeRefreshTimer = import ./lib/store-refresh-timer.nix { }; atomicWriteSecret = import ./lib/atomic-write-secret.nix { }; refreshConsumer = import ./lib/refresh-consumer.nix { }; + rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; }; in { # What the store IS from any hive's point of view, the name it answers on, @@ -728,6 +729,8 @@ in systemd.timers.${forwarderFetchUnit} = storeRefreshTimer "re-read the metrics store forwarder's OIDC client secret from the swarm secret store"; + systemd.services."container@${cfg.machine}" = rootfsCheck.warnOnly cfg.machine; + containers.${cfg.machine} = { autoStart = true; ephemeral = false; diff --git a/nix/module-eval/core-toggle.nix b/nix/module-eval/core-toggle.nix index eed0f856..668e41fa 100644 --- a/nix/module-eval/core-toggle.nix +++ b/nix/module-eval/core-toggle.nix @@ -19,6 +19,7 @@ let ; }) hive + hiveWith hiveWithController runGroup forceServiceConfigs @@ -73,6 +74,13 @@ let gateway.openFirewall = true; }; + # A container the operator runs that hyperhive doesn't own, next to one it + # does (the classifier's). + withForeignContainer = hiveWith [ { containers.operator-box.config = { }; } ] { + deploy.bao.enable = true; + deploy.review-classifier.enable = true; + }; + # On the store's host, which mints the runner's store identity. withCi = hive { deploy.bao.enable = true; @@ -1498,35 +1506,76 @@ let && (sc.ProtectKernelModules or false); } { - # Every `container@` instance starts through the template's drop-in, the - # agents' and the swarm services' alike, so a host with containers has it - # whether or not it runs a hive. A host with none has no `container@` - # unit and gets no drop-in. Only the rendered line is pinned: eval cannot - # show where systemd reads the file from. The misspelled name is the - # control that the lookup finds nothing when nothing is there. - name = "the container@ template drop-in runs the root check on every host with containers and on no other"; + # Hyperhive's own declarative containers each run the root check from + # their own unit, warn-only. Every container in these fixtures is one of + # hyperhive's, so the names are read off the fixture: a container added + # without the check breaks the list, which is the point. The unit's own + # pre-start script is not a `hive-priv` entry, so the count is the check + # alone. + name = "every hyperhive container runs the root check warn-only from its own unit, and no hyperhive container is left out"; ok = let - dropinText = - name: m: + checks = + m: name: let - hits = lib.filter (p: (p.name or "") == name) m.systemd.packages; + pre = m.systemd.services."container@${name}".serviceConfig.ExecStartPre or [ ]; in - if builtins.length hits == 1 then (builtins.head hits).text else ""; - runsCheck = - name: m: - let - want = "ExecStartPre=${m.services.hyperhive.c0re.package}/bin/hive-priv check-rootfs %i"; - lines = lib.splitString "\n" (dropinText name m); - in - lib.elem (builtins.unsafeDiscardStringContext want) (map builtins.unsafeDiscardStringContext lines); - check = "hyperhive-container-rootfs-check"; + builtins.filter (lib.hasInfix "hive-priv") ( + map (p: builtins.unsafeDiscardStringContext (toString p)) pre + ); + carries = + m: name: + checks m name == [ + (builtins.unsafeDiscardStringContext "${m.services.hyperhive.c0re.package}/bin/hive-priv check-rootfs --warn-only ${name}") + ]; + names = lib.attrNames allLocal.containers; in - runsCheck check allLocal - && runsCheck check shared.servicesHere - && !(runsCheck "${check}-misspelled" allLocal) - && !(bare.systemd.units ? "container@.service") - && !(runsCheck check bare); + names == [ + "hive-forge" + "hive-matrix" + "swarm-attic" + "swarm-authelia" + "swarm-bao" + "swarm-builder" + "swarm-claude-login" + "swarm-grafana" + "swarm-nats" + "swarm-notify-router" + "swarm-otel" + "swarm-review-classifier" + "swarm-victorialogs" + "swarm-victoriametrics" + ] + && lib.all (carries allLocal) names + && withCi.containers ? swarm-ci-runner + && lib.all (carries withCi) (lib.attrNames withCi.containers) + # Control: a name with no unit finds nothing. + && checks allLocal "swarm-nope" == [ ]; + } + { + # The operator's other containers are not hyperhive's to touch: a + # `containers.` it declares gets no check, while the classifier + # beside it does. The positive arm is what makes the absence mean + # something: the same lookup finds the classifier's. + name = "a container hyperhive does not declare gets no root check"; + ok = + let + checks = + name: + builtins.filter (lib.hasInfix "hive-priv") ( + map (p: builtins.unsafeDiscardStringContext (toString p)) ( + withForeignContainer.systemd.services."container@${name}".serviceConfig.ExecStartPre or [ ] + ) + ); + in + lib.attrNames withForeignContainer.containers == [ + "operator-box" + "swarm-bao" + "swarm-review-classifier" + ] + && lib.length (checks "swarm-review-classifier") == 1 + && lib.length (checks "swarm-bao") == 1 + && checks "operator-box" == [ ]; } ] # Starting a unit that stays `active (exited)` is a no-op, so a timer aimed diff --git a/nix/module-eval/lib.nix b/nix/module-eval/lib.nix index b9f84ac5..fa10706f 100644 --- a/nix/module-eval/lib.nix +++ b/nix/module-eval/lib.nix @@ -42,8 +42,8 @@ let # nixpkgs. `read-only.nix` makes any host module that defines top-level # `nixpkgs.config`/`overlays`/`hostPlatform` fail the fixture's eval instead # of being ignored. - hive = - extra: + hiveWith = + modules: extra: (nixosSystem { modules = [ (pkgs.path + "/nixos/modules/misc/nixpkgs/read-only.nix") @@ -70,9 +70,13 @@ let swarm.hives.h1.domain = "h1.t.local"; } extra; } - ]; + ] + ++ modules; }).config; + # `hiveWith` with no extra modules: the host a fixture normally is. + hive = hiveWith [ ]; + # `hive` with the hive-controller tier on (hive-c0re, the agent containers, # `deploy.hive-controller.*`). Only a fixture whose claim reads that tier # takes this; every other claim is about the swarm services and evaluates @@ -299,7 +303,7 @@ let builtins.foldl' (acc: v: builtins.seq v acc) true vals; in { - inherit hive; + inherit hive hiveWith; inherit hiveWithController; inherit agent; inherit agentWith; diff --git a/nix/vm-tests/agent-ephemeral-rootfs.nix b/nix/vm-tests/agent-ephemeral-rootfs.nix index a6c685e3..74a651aa 100644 --- a/nix/vm-tests/agent-ephemeral-rootfs.nix +++ b/nix/vm-tests/agent-ephemeral-rootfs.nix @@ -13,10 +13,10 @@ # forwarder (nix/agent-modules/otel.nix) reads it, and the host can read it; # - a package installed into the user's nix profile is still there, and on # PATH, after a restart; -# - a link planted at `root` in the root template refuses the next start through -# the `ExecStartPre=` check in the `container@` template drop-in -# (nix/host-modules/container-rootfs-check.nix), leaves the link's target -# untouched, and the start goes through once the link is gone. +# - a link planted at `root` in the template refuses the next start through +# the limits drop-in's `ExecStartPre=` check (whose build has a GC root), +# leaves the link's target untouched, and the start goes through once the +# link is gone. # # The agent's journal forwarder itself (an OTLP push to the hive's collector) is # not booted here; the module-eval suite covers its wiring. @@ -69,7 +69,6 @@ pkgs.testers.runNixOSTest { { imports = [ hiveStubs - ../host-modules/container-rootfs-check.nix ../host-modules/hive-priv.nix ]; virtualisation.memorySize = 2048; @@ -234,10 +233,21 @@ pkgs.testers.runNixOSTest { with subtest("a link planted in the root refuses the start and nothing is touched"): stop() - # The pre-start check is a drop-in of the `container@` template, so the - # unit carries it with no per-container drop-in written. - unit = host.succeed(f"systemctl cat container@{MACHINE}") - assert "bin/hive-priv check-rootfs %i\n" in unit, unit + # What hive-c0re does besides WriteNspawnFlags: the limits drop-in, + # which carries the pre-start check, and the runtime dir its condition + # names. + host.succeed(f"mkdir -p /run/hyperhive/agents/{AGENT}") + for req in [ + {"op": "write_resource_limits", "container": MACHINE, "memory_max": "1G", "cpu_quota": "200%"}, + {"op": "daemon_reload"}, + ]: + out = host.succeed(f"echo '{json.dumps(req)}' | socat - UNIX-CONNECT:/run/hive/priv.sock") + assert json.loads(out.strip().splitlines()[-1])["ok"] is True, out + dropin = host.succeed(f"cat /run/systemd/system/container@{MACHINE}.service.d/hyperhive-limits.conf") + assert "ExecStartPre=/nix/store/" in dropin and f"check-rootfs {MACHINE}\n" in dropin, dropin + # The build the drop-in names is rooted against garbage collection. + build = "/".join(dropin.split("ExecStartPre=")[1].split()[0].split("/")[:4]) + assert host.succeed(f"readlink /nix/var/nix/gcroots/hyperhive-check-rootfs/{MACHINE}").strip() == build, build # `root` is a link to a directory outside the root, which the start # script would `chmod 0700` as host root.