From 81439fa7105272c7846aee0aaf25be4b37aac10b Mon Sep 17 00:00:00 2001 From: atlas Date: Wed, 28 Oct 2026 19:15:26 +0100 Subject: [PATCH] run agent containers on an ephemeral root, salvaging the old one first hive-priv copies a stopped container's root to the agent's work dir and renames it aside before passing nspawn --ephemeral; the agent's nix state moves to the persistent profiles bind. Adds the agent-ephemeral-rootfs vm test and a module-eval case for the nix state redirect. --- docs/agent-lifecycle/persistence.md | 51 +- docs/process/gotchas.md | 7 +- docs/scheduler/ci.md | 3 +- docs/scheduler/observability.md | 2 +- docs/trust-boundary/security.md | 2 +- flake.nix | 8 +- hive-agent/prompts/system.md | 2 +- hive-priv-sock/src/lib.rs | 5 + hive-priv/src/container_config.rs | 94 ++- hive-priv/src/main.rs | 10 +- hive-priv/src/rootfs.rs | 842 ++++++++++++++++++++++++ nix/agent-modules/otel.nix | 2 +- nix/agent-modules/user.nix | 20 + nix/module-eval/agent-user.nix | 13 + nix/vm-tests.nix | 3 +- nix/vm-tests/agent-ephemeral-rootfs.nix | 213 ++++++ 16 files changed, 1238 insertions(+), 39 deletions(-) create mode 100644 hive-priv/src/rootfs.rs create mode 100644 nix/vm-tests/agent-ephemeral-rootfs.nix diff --git a/docs/agent-lifecycle/persistence.md b/docs/agent-lifecycle/persistence.md index 1838fed4..3043a128 100644 --- a/docs/agent-lifecycle/persistence.md +++ b/docs/agent-lifecycle/persistence.md @@ -341,6 +341,53 @@ Under `/var/lib/hyperhive/agents//`: in the per-agent disk metric (`measure_agent_disk`) together with `state/` and the container rootfs. The dashboard doesn't serve it. +### The container root (per agent) + +An agent container starts with `systemd-nspawn --ephemeral` +(`hive-priv`'s `WriteNspawnFlags` passes the flag). nspawn runs it on a +snapshot of `/var/lib/nixos-containers/h-`, and the snapshot vanishes +at stop. The container loses anything the agent writes outside a bind: `/home/` +apart from `.claude/`, `/etc`, `/var`, `/root`, `/tmp`. What persists is +`state/`, `work/`, `harness/`, `claude/` (at `~/.claude`), the per-container +`/nix/var/nix/profiles` and `/nix/var/nix/gcroots` binds, and the sockets dir. +`/home/` holds caches and nothing the agent should keep: worktrees, +clones and builds go in `work/`. + +The nix profile needs one extra step. nix keeps a non-root user's profile in +`~/.local/state/nix`, so the agent's tmpfiles rules +(`nix/agent-modules/user.nix`) point that directory into +`/nix/var/nix/profiles/per-user/` (the persistent bind) and recreate +`~/.nix-profile` at every boot. Packages the agent installs with `nix-env` or +`nix profile` survive a restart, and go with the container on `destroy`. + +**The one-time salvage.** The first start of an agent whose root has content +runs `hive-priv/src/rootfs.rs` before hive-priv passes `--ephemeral`, from the +`WriteNspawnFlags` that precedes every start, while the container is down: + +1. Everything in the root except the cache paths (`tmp`, `var/cache`, + `var/log/journal`, `var/lib/weston`, `var/lib/opentelemetry-collector`) is + copied to `work/salvaged-rootfs-/`, owned by the agent, with + modes and mtimes kept. The agent finds its old `/home/` there. +2. hive-priv copies the agent's nix state in the old home to the persistent profile bind, following no link in the path (the container can write there). +3. hive-priv renames the old root to `h-.pre-ephemeral-` and + logs the path. Nothing deletes it; the operator removes it when ready. +4. A fresh template root takes its place, carrying `etc/machine-id` (the old + id, so the journal directory and DHCP identity hold) and the marker + `etc/hyperhive-ephemeral-root`. + +The salvage runs again only if the template holds content outside the cache +paths, which happens when a hive without this change started the container +persistently in between; the new copy gets a numbered directory beside the +first. A failed salvage leaves the root where it was and the container starts +without the flag (hive-priv logs the error), so a failure costs the switch, +never data. + +The journal still ships. The agent's collector (`nix/agent-modules/otel.nix`) +reads the guest's own `/var/log/journal` and pushes it over OTLP, so no +journal link to the host takes part; nspawn skips `--link-journal=try-guest` +in ephemeral mode without failing. The journal vanishes at stop, and with it +the lines written after the last push. + ### Cross-agent access to state An agent holding the `manage_root_agent` permission gets every other @@ -562,7 +609,9 @@ container lifetime: `${homeDir}/.claude`.** Marker (`/var/lib/hive-agent-user-migrated`) guards single-shot; it's only written once there's nothing left to migrate, so a `cp` - failure leaves it absent and the next boot retries. + failure leaves it absent and the next boot retries. The root is + ephemeral, so the marker vanishes at every stop and the step + re-runs; with no `/root/.claude` it only writes the marker again. `cp -an` (no-clobber) so any pre-existing files at the new location win — never blow over data already there, and so a retry after a partial copy is as safe as the first attempt. diff --git a/docs/process/gotchas.md b/docs/process/gotchas.md index 632eccb2..747eabec 100644 --- a/docs/process/gotchas.md +++ b/docs/process/gotchas.md @@ -211,9 +211,10 @@ alongside the claude dir. `/agents//work` (RW, owned by the agent user). It's the place for git worktrees and build dirs: a claude started under `state/` walks up into `state/AGENTS.md` and takes on the agent's identity, and one -started under `work/` doesn't. `/home/` lives in the container -rootfs, which `destroy` deletes; `work/` survives `destroy`/recreate -and goes on `--purge`. The per-agent disk metric counts it; the +started under `work/` doesn't. The container rootfs is ephemeral +(nspawn `--ephemeral`), so `/home/` is empty after every restart: +anything the agent keeps goes in `work/` or `state/`. `work/` survives +`destroy`/recreate and goes on `--purge`. The per-agent disk metric counts it; the dashboard doesn't serve it. ## Networking & ports diff --git a/docs/scheduler/ci.md b/docs/scheduler/ci.md index 62a82ddf..0ab65698 100644 --- a/docs/scheduler/ci.md +++ b/docs/scheduler/ci.md @@ -119,7 +119,8 @@ builds all of them nightly at 02:30; a failing test fails that run. To run one by hand, start the workflow from the forge UI (Actions, vm-nightly, Run workflow, optional `test` input), or run `nix build .#vmTests.x86_64-linux.` on a host with KVM. Current tests: -`hive-squid`. +`agent-ephemeral-rootfs` (the switch of an agent container to an ephemeral +root, through the real hive-priv unit) and `hive-squid`. ## Configuration reference diff --git a/docs/scheduler/observability.md b/docs/scheduler/observability.md index 5f836a88..6409b04a 100644 --- a/docs/scheduler/observability.md +++ b/docs/scheduler/observability.md @@ -340,7 +340,7 @@ exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource | `container.memory.usage` | `By` | gauge | `memory.current` | | `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) | | `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) | -| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs, measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) | +| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + the root template (the live ephemeral layer isn't counted), measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) | | `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) | | `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable | | `hyperhive.agent.closure_pullable` | `1` | gauge | `1` when this hive can fetch the agent's latest closure from the swarm's binary cache and verify its signature, else `0` with a `reason` label (`no-record`, `rev-mismatch`, `not-cached`, `no-credentials`, `fetch-failed`, `bad-signature`; see [the builder docs](../swarm/builder.md#what-hives-do-with-the-cache)). Omitted before the first probe pass, every 5 minutes | diff --git a/docs/trust-boundary/security.md b/docs/trust-boundary/security.md index 06e7b1c2..7f076c1d 100644 --- a/docs/trust-boundary/security.md +++ b/docs/trust-boundary/security.md @@ -261,7 +261,7 @@ known operations; there is no arbitrary command pass-through: | `DestroyContainer` | `nixos-container destroy ` | | `ListContainers` | `nixos-container list` | | `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) | -| `WriteNspawnFlags` | write `/etc/nixos-containers/.conf` (bind-mount list + network isolation vars) | +| `WriteNspawnFlags` | write `/etc/nixos-containers/.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted | | `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else | | `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container | | `RemoveServiceDropin` | remove `container@.service.d/` drop-in on destroy | diff --git a/flake.nix b/flake.nix index d619917a..a78e6ec2 100644 --- a/flake.nix +++ b/flake.nix @@ -228,7 +228,13 @@ # Tests that boot a NixOS VM and need KVM. Not `checks`, so `nix flake # check` and the per-PR CI job never build them; # `.forgejo/workflows/vm-nightly.yml` does. - vmTests = forAllSystems ({ pkgs, ... }: import ./nix/vm-tests.nix { inherit pkgs; }); + vmTests = forAllSystems ( + { pkgs, system, ... }: + import ./nix/vm-tests.nix { + inherit pkgs; + hivePriv = self.packages.${system}.hive-priv; + } + ); checks = forAllSystems ( { diff --git a/hive-agent/prompts/system.md b/hive-agent/prompts/system.md index 5d63c498..2d4bdccc 100644 --- a/hive-agent/prompts/system.md +++ b/hive-agent/prompts/system.md @@ -33,7 +33,7 @@ Durable knowledge: - write to `/agents/{label}/state/notes.md` (free-form) or any other path under `/agents/{label}/state/`. That directory is bind-mounted from the host and persists across container destroy/recreate — claude's `--continue` session only carries short-term context, but `/agents/{label}/state/` is forever. Read it back at the start of relevant turns to remember things across resets. -**Work dir**: `/agents/{label}/work/` is the place for git worktrees, clones, build trees and other bulky scratch. It is bind-mounted from the host, persists across container destroy/recreate and is backed up. Keep `state/` for notes and small durable files. Don't keep work under `/home`, `/tmp` or other container paths — they are outside what's backed up. Start subagents with their `dir` under `work/` (that is also the default): a claude started under `state/` walks up into `state/AGENTS.md` and takes on your identity, one under `work/` doesn't. +**Work dir**: `/agents/{label}/work/` is the place for git worktrees, clones, build trees and other bulky scratch. It is bind-mounted from the host, persists across container destroy/recreate and is backed up. Keep `state/` for notes and small durable files. Don't keep work under `/home`, `/tmp` or other container paths: the container discards them every time it stops. Start subagents with their `dir` under `work/` (that is also the default): a claude started under `state/` walks up into `state/AGENTS.md` and takes on your identity, one under `work/` doesn't. Claude session (OAuth credentials) lives at `/root/.claude/` and persists across restarts. diff --git a/hive-priv-sock/src/lib.rs b/hive-priv-sock/src/lib.rs index 9524b49f..d8e066fd 100644 --- a/hive-priv-sock/src/lib.rs +++ b/hive-priv-sock/src/lib.rs @@ -330,6 +330,11 @@ pub enum PrivRequest { /// Always writes `PRIVATE_NETWORK=1` + veth wiring: isolation is the only /// mode, so there is no request shape that yields a container sharing the /// host's network namespace. + /// + /// Also decides whether the container runs on an ephemeral root. When it is + /// stopped and its root holds anything beyond caches, the root is copied to + /// the agent's `work/` and renamed aside first, then `--ephemeral` is added + /// to the flags; see `hive-priv/src/rootfs.rs`. WriteNspawnFlags { container: String, binds: Vec, diff --git a/hive-priv/src/container_config.rs b/hive-priv/src/container_config.rs index 6b79966e..2984ac1d 100644 --- a/hive-priv/src/container_config.rs +++ b/hive-priv/src/container_config.rs @@ -27,7 +27,7 @@ pub(crate) fn handle_write_nspawn_flags( load_credentials: &[CredentialMount], hive_facts: &HiveFacts, ) -> Result<(String, String)> { - validate_container_system_name(container)?; + let agent = validate_container_system_name(container)?; for bind in binds { validate_bind_path(&bind.host_path)?; validate_bind_path(&bind.container_path)?; @@ -40,7 +40,24 @@ pub(crate) fn handle_write_nspawn_flags( validate_bind_path(&cred.host_path)?; } let hive_facts = render_hive_facts(hive_facts)?; - write_nspawn_flags(container, binds, isolation, load_credentials, &hive_facts)?; + // A root that cannot be salvaged keeps its persistent start: the flag is + // only ever passed for a root the salvage has dealt with. + let ephemeral = crate::rootfs::prepare(container, agent).unwrap_or_else(|e| { + tracing::error!( + %container, + error = %format!("{e:#}"), + "root salvage failed; the container starts with its persistent root" + ); + false + }); + write_nspawn_flags( + container, + binds, + isolation, + load_credentials, + &hive_facts, + ephemeral, + )?; Ok((String::new(), String::new())) } @@ -267,6 +284,7 @@ fn write_nspawn_flags( isolation: &NetworkIsolation, load_credentials: &[CredentialMount], hive_facts: &str, + ephemeral: bool, ) -> Result<()> { use std::fmt::Write as _; use std::os::unix::fs::MetadataExt as _; @@ -309,25 +327,7 @@ fn write_nspawn_flags( out.push_str("LOCAL_ADDRESS6=\n"); let _ = writeln!(out, "HOST_BRIDGE={}", iso.bridge); } - let mut flags: Vec = binds - .iter() - .map(|b| { - let flag = if b.read_only { "--bind-ro" } else { "--bind" }; - format!("{flag}={}:{}", b.host_path, b.container_path) - }) - .collect(); - flags.extend(git_overlay_flags(binds)); - // Credential forwarding: nspawn loads each host secret into the - // container's credential store under ``; inner units inherit it - // via `LoadCredential=`. Validated (name charset + bind-path - // rules) in handle_write_nspawn_flags above. - for cred in load_credentials { - flags.push(format!( - "--load-credential={}:{}", - cred.name, cred.host_path - )); - } - let flags_joined = flags.join(" "); + let flags_joined = nspawn_flags(binds, load_credentials, ephemeral).join(" "); let _ = writeln!(out, "EXTRA_NSPAWN_FLAGS=\"{flags_joined}\""); // Keeps the owner and mode `nixos-container` created the file with; this // helper only rewrites its lines. @@ -353,6 +353,38 @@ fn write_nspawn_flags( Ok(()) } +/// The flags `EXTRA_NSPAWN_FLAGS` carries, in the order nspawn needs them. +/// +/// `--ephemeral` runs the container on a snapshot of its root and drops the +/// snapshot at stop; `crate::rootfs` has made the root fit for that before +/// the flag is asked for. The git overlays and credentials follow the binds +/// they refer to. +fn nspawn_flags( + binds: &[BindMount], + load_credentials: &[CredentialMount], + ephemeral: bool, +) -> Vec { + let mut flags: Vec = ephemeral + .then(|| "--ephemeral".to_owned()) + .into_iter() + .collect(); + flags.extend(binds.iter().map(|b| { + let flag = if b.read_only { "--bind-ro" } else { "--bind" }; + format!("{flag}={}:{}", b.host_path, b.container_path) + })); + flags.extend(git_overlay_flags(binds)); + // Credential forwarding: nspawn loads each host secret into the + // container's credential store under ``; inner units inherit it + // via `LoadCredential=`. Validated (name charset + bind-path + // rules) in handle_write_nspawn_flags. + flags.extend( + load_credentials + .iter() + .map(|cred| format!("--load-credential={}:{}", cred.name, cred.host_path)), + ); + flags +} + /// Filename of the in-container DNS marker, in the container's own `/etc`. const BRIDGE_DNS_MARKER: &str = "hyperhive-bridge-dns"; @@ -360,7 +392,7 @@ const BRIDGE_DNS_MARKER: &str = "hyperhive-bridge-dns"; const HIVE_FACTS_FILE: &str = "hyperhive-hive-facts"; /// The container's `/` on the host. -fn container_rootfs(container: &str) -> PathBuf { +pub(crate) fn container_rootfs(container: &str) -> PathBuf { PathBuf::from(format!("/var/lib/nixos-containers/{container}")) } @@ -469,7 +501,7 @@ fn write_bridge_dns_marker_in(rootfs: &Path, gateway_ip: &str) -> Result<()> { /// materialises the rootfs on the first start), so this creates it first; the /// dir and the file persist through that start. Mode 0644: nothing written /// here is a secret. -fn write_container_etc_file_in(rootfs: &Path, name: &str, contents: &str) -> Result<()> { +pub(crate) fn write_container_etc_file_in(rootfs: &Path, name: &str, contents: &str) -> Result<()> { use std::os::unix::fs::OpenOptionsExt as _; let etc_path = rootfs.join("etc"); @@ -510,7 +542,7 @@ fn write_container_etc_file_in(rootfs: &Path, name: &str, contents: &str) -> Res mod tests { use super::{ BindMount, HiveFacts, NetworkIsolation, git_overlay_flags, handle_write_nspawn_flags, - limits_dropin_body, publish_file, remove_service_dropin, render_hive_facts, + limits_dropin_body, nspawn_flags, publish_file, remove_service_dropin, render_hive_facts, validate_container_system_name, write_bridge_dns_marker_in, write_resource_limits, }; use crate::shared::test_support::{entries, scratch}; @@ -909,6 +941,20 @@ mod tests { } } + /// `--ephemeral` is asked for, not assumed: present exactly when the root has + /// been dealt with, ahead of the binds it applies to. The binds are the + /// control that the lookup finds what is there. + #[test] + fn ephemeral_is_passed_only_when_asked() { + let binds = [bind("/agents/atlas/work")]; + let with = nspawn_flags(&binds, &[], true); + assert_eq!(with[0], "--ephemeral", "{with:?}"); + assert!(with.iter().any(|f| f.starts_with("--bind-ro=")), "{with:?}"); + let without = nspawn_flags(&binds, &[], false); + assert!(!without.iter().any(|f| f == "--ephemeral"), "{without:?}"); + assert_eq!(without.len() + 1, with.len()); + } + /// The swarm's forge and matrix containers are not the hive's: every /// request that names a container as a string refuses them before /// touching the filesystem. Control: an agent container name passes. diff --git a/hive-priv/src/main.rs b/hive-priv/src/main.rs index c6afa11b..e09c4793 100644 --- a/hive-priv/src/main.rs +++ b/hive-priv/src/main.rs @@ -19,16 +19,18 @@ //! **Layout**: this file holds the socket loop and the one-arm-per-op //! dispatch table. Each op family's handlers and validation live in their //! own module: [`container`] (lifecycle), [`container_config`] (nspawn flags, -//! systemd drop-ins), [`storage`] (btrfs subvolumes, quotas, snapshots, disk -//! measurement), [`files`] (pause marker, socket dir, legacy tmpfiles), [`infra`] (gateway nginx, -//! infra containers) and [`closure`] (downloads from the swarm cache). Helpers -//! more than one family uses are in [`shared`]. +//! systemd drop-ins), [`rootfs`] (the salvage of a container's root before it +//! turns ephemeral), [`storage`] (btrfs subvolumes, quotas, snapshots, disk +//! measurement), [`files`] (pause marker, socket dir, legacy tmpfiles), +//! [`infra`] (gateway nginx, infra containers) and [`closure`] (downloads from +//! the swarm cache). Helpers more than one family uses are in [`shared`]. mod closure; mod container; mod container_config; mod files; mod infra; +mod rootfs; mod shared; mod storage; diff --git a/hive-priv/src/rootfs.rs b/hive-priv/src/rootfs.rs new file mode 100644 index 00000000..870c3ef7 --- /dev/null +++ b/hive-priv/src/rootfs.rs @@ -0,0 +1,842 @@ +//! The agent container's non-persistent root. +//! +//! Agent containers start with `systemd-nspawn --ephemeral`: nspawn runs the +//! container on a throwaway snapshot of `/var/lib/nixos-containers/`, so +//! anything written outside the binds is gone at stop. [`prepare`] runs ahead +//! of every `WriteNspawnFlags` and decides whether the flag may be passed. +//! +//! A root that holds anything beyond what the start path itself writes is +//! first salvaged. Everything outside [`CACHE`] is copied into +//! `/work/salvaged-rootfs-/`, owned by the agent, the old +//! root is renamed to `.pre-ephemeral-` (never deleted), and a +//! fresh template root takes its place. The template is what nspawn snapshots +//! on every start, so it must stay small and free of the previous contents. +//! The template carries [`MARKER`]; it is written last, so a crash anywhere +//! earlier leaves the old root in place and the next call tries again. +//! +//! Only a stopped container is touched: a running one keeps whatever flag its +//! root already warrants. + +use std::os::unix::fs::{MetadataExt as _, PermissionsExt as _, lchown, symlink}; +use std::path::{Path, PathBuf}; + +use anyhow::{Context as _, Result, bail}; +use hive_priv_sock::AGENT_STATE_ROOT; + +use crate::container_config::{container_rootfs, write_container_etc_file_in}; + +/// The per-container nix profile bind on the host, which persists across starts. +/// What nix keeps for the agent user is carried here; see [`carry_nix_state`]. +const PROFILES_ROOT: &str = "/nix/var/nix/profiles/per-container"; + +/// Written into a template root once it is ready to be snapshotted. +const MARKER: &str = "hyperhive-ephemeral-root"; + +/// Files in `etc/` that the start path writes into every root, the template +/// included. They are not content: the root counts as clean with only these. +const TEMPLATE_FILES: [&str; 6] = [ + MARKER, + "hyperhive-bridge-dns", + "hyperhive-hive-facts", + "machine-id", + "os-release", + "resolv.conf", +]; + +/// Paths relative to the root that the salvage skips: the guest journal, its +/// forwarder's read cursor, weston's state and scratch space. The first start +/// of a container rebuilds all of them. +const CACHE: [&str; 5] = [ + "tmp", + "var/cache", + "var/log/journal", + "var/lib/weston", + "var/lib/opentelemetry-collector", +]; + +/// What [`rotate`] did. +#[derive(Debug, PartialEq, Eq)] +pub(crate) enum Outcome { + /// Nothing outside [`CACHE`] was in the root; nothing was moved. + Clean, + /// The root's content is copied to `salvage`, and the root itself kept at + /// `old_root`. + Salvaged { salvage: PathBuf, old_root: PathBuf }, +} + +/// Whether `container` may be started with `--ephemeral`, salvaging its root +/// first when it holds content and the container is stopped. +/// +/// # Errors +/// Returns an error when the unit state cannot be read or the salvage fails. +/// Nothing is deleted on that path, and the caller starts the container +/// without the flag. +pub(crate) fn prepare(container: &str, agent: &str) -> Result { + let root = container_rootfs(container); + if !unit_is_stopped(container)? { + return Ok(is_regular(&root.join("etc").join(MARKER))); + } + let work = Path::new(AGENT_STATE_ROOT).join(agent).join("work"); + let profiles = Path::new(PROFILES_ROOT).join(container); + match rotate(&root, agent, &work, &profiles, &today())? { + Outcome::Clean => {} + Outcome::Salvaged { salvage, old_root } => tracing::warn!( + %container, + salvage = %salvage.display(), + old_root = %old_root.display(), + "root salvaged before the first ephemeral start; the old root is kept, remove it by hand" + ), + } + Ok(true) +} + +/// Salvage `root` if it holds content, then leave a marked template in its +/// place. `work` is the agent's work dir on the host, `profiles` the host side of +/// the container's `/nix/var/nix/profiles`; `date` is `YYYYMMDD`. +fn rotate(root: &Path, agent: &str, work: &Path, profiles: &Path, date: &str) -> Result { + let outcome = if has_content(root)? { + let (uid, gid) = agent_owner(root, agent, work)?; + let machine_id = read_machine_id(root); + let salvage = fresh_dir(work, &format!("salvaged-rootfs-{date}"))?; + lchown(&salvage, Some(uid), Some(gid)) + .with_context(|| format!("chown {}", salvage.display()))?; + copy_tree(root, &salvage, Path::new(""), uid, gid) + .with_context(|| format!("copy {} to {}", root.display(), salvage.display()))?; + carry_nix_state(root, agent, profiles, uid, gid)?; + let old_root = free_sibling(root, &format!("pre-ephemeral-{date}"))?; + std::fs::rename(root, &old_root) + .with_context(|| format!("move {} to {}", root.display(), old_root.display()))?; + std::fs::create_dir(root).with_context(|| format!("create {}", root.display()))?; + if let Some(id) = machine_id { + write_container_etc_file_in(root, "machine-id", &format!("{id}\n"))?; + } + Outcome::Salvaged { salvage, old_root } + } else { + Outcome::Clean + }; + ensure_machine_id(root)?; + write_container_etc_file_in(root, MARKER, "")?; + Ok(outcome) +} + +/// Whether anything but directories, [`TEMPLATE_FILES`] and [`CACHE`] is under +/// `root`. A missing root has none. +fn has_content(root: &Path) -> Result { + fn walk(dir: &Path, rel: &Path) -> Result { + let entries = match std::fs::read_dir(dir) { + Ok(entries) => entries, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(false), + Err(e) => return Err(e).with_context(|| format!("read {}", dir.display())), + }; + for entry in entries { + let entry = entry.with_context(|| format!("read {}", dir.display()))?; + let rel = rel.join(entry.file_name()); + if is_cache(&rel) { + continue; + } + let kind = entry + .file_type() + .with_context(|| format!("stat {}", entry.path().display()))?; + let is_template = TEMPLATE_FILES + .iter() + .any(|name| rel == Path::new("etc").join(name)); + if kind.is_dir() { + if walk(&entry.path(), &rel)? { + return Ok(true); + } + } else if !is_template { + return Ok(true); + } + } + Ok(false) + } + walk(root, Path::new("")) +} + +fn is_cache(rel: &Path) -> bool { + CACHE.iter().any(|c| rel == Path::new(c)) +} + +/// The uid and gid the salvage is handed to: the owner of the agent's home in +/// the old root, else of its work dir. Never root. +fn agent_owner(root: &Path, agent: &str, work: &Path) -> Result<(u32, u32)> { + let home = real_dir(root, &["home", agent]); + for dir in [home, real_dir(work, &[])].into_iter().flatten() { + if let Ok(meta) = std::fs::symlink_metadata(&dir) + && meta.uid() != 0 + { + return Ok((meta.uid(), meta.gid())); + } + } + bail!( + "agent {agent}: neither its home in {} nor {} has a non-root owner", + root.display(), + work.display() + ) +} + +/// `base` joined with `parts`, if every component is a real directory. The +/// container could have made any of them a link, and this runs as root. +fn real_dir(base: &Path, parts: &[&str]) -> Option { + let mut path = base.to_path_buf(); + if !std::fs::symlink_metadata(&path).ok()?.is_dir() { + return None; + } + for part in parts { + path.push(part); + if !std::fs::symlink_metadata(&path).ok()?.is_dir() { + return None; + } + } + Some(path) +} + +/// nix keeps a non-root user's profile under `~/.local/state/nix`, which the +/// agent's tmpfiles rules (`nix/agent-modules/user.nix`) point at +/// `per-user/` in `profiles`. A root that was lived in has the real +/// directory there: copy it across, so the agent's installed packages and the +/// GC roots that name them are still valid after the switch. A target that +/// holds anything is left alone; so is the whole copy when a component of the +/// target path is not a real directory, as the container can write `profiles`. +fn carry_nix_state(root: &Path, agent: &str, profiles: &Path, uid: u32, gid: u32) -> Result<()> { + let Some(src) = real_dir(root, &["home", agent, ".local", "state", "nix"]) else { + return Ok(()); + }; + let Some(target) = own_dir(profiles, &["per-user", agent], uid, gid)? else { + tracing::warn!( + profiles = %profiles.display(), + "a link or file where the nix state goes; the agent's nix state is in the salvage only" + ); + return Ok(()); + }; + // The tmpfiles rules create `target` empty on a boot of the new module; + // anything in it is the agent's newer state. + if std::fs::read_dir(&target) + .with_context(|| format!("read {}", target.display()))? + .next() + .is_some() + { + return Ok(()); + } + copy_tree(&src, &target, Path::new(""), uid, gid) + .with_context(|| format!("copy {} to {}", src.display(), target.display())) +} + +/// `base` joined with `parts`, creating each missing component (the last one +/// owned by `uid:gid`), or `None` when `base` or an existing component is not a +/// real directory. The container could have replaced any of them by a link, and +/// this runs as root. +fn own_dir(base: &Path, parts: &[&str], uid: u32, gid: u32) -> Result> { + let mut path = base.to_path_buf(); + if !std::fs::symlink_metadata(&path).is_ok_and(|m| m.is_dir()) { + return Ok(None); + } + for (i, part) in parts.iter().enumerate() { + path.push(part); + match std::fs::symlink_metadata(&path) { + Ok(meta) if meta.is_dir() => {} + Ok(_) => return Ok(None), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => { + std::fs::create_dir(&path).with_context(|| format!("create {}", path.display()))?; + if i + 1 == parts.len() { + lchown(&path, Some(uid), Some(gid)) + .with_context(|| format!("chown {}", path.display()))?; + } + } + Err(e) => return Err(e).with_context(|| format!("stat {}", path.display())), + } + } + Ok(Some(path)) +} + +/// Create `parent/base`, or `parent/base-2`, ... when taken. Created, never +/// reused: `mkdir` does not follow a symlink an agent left under that name. +fn fresh_dir(parent: &Path, base: &str) -> Result { + for n in 1.. { + let path = if n == 1 { + parent.join(base) + } else { + parent.join(format!("{base}-{n}")) + }; + match std::fs::create_dir(&path) { + Ok(()) => return Ok(path), + Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => {} + Err(e) => return Err(e).with_context(|| format!("create {}", path.display())), + } + } + unreachable!("the loop above returns or continues") +} + +/// `.`, or `.-2`, ... the first name nothing uses. +fn free_sibling(root: &Path, suffix: &str) -> Result { + let name = root + .file_name() + .and_then(|n| n.to_str()) + .with_context(|| format!("{} has no usable name", root.display()))?; + for n in 1.. { + let candidate = if n == 1 { + format!("{name}.{suffix}") + } else { + format!("{name}.{suffix}-{n}") + }; + let path = root.with_file_name(candidate); + if std::fs::symlink_metadata(&path).is_err() { + return Ok(path); + } + } + unreachable!("the loop above returns or continues") +} + +/// Copy everything under `src` except [`CACHE`] into `dst`, owned by +/// `uid:gid`, with modes and mtimes kept. Sockets, fifos and device nodes are +/// skipped; hard links become separate copies. +fn copy_tree(src: &Path, dst: &Path, rel: &Path, uid: u32, gid: u32) -> Result<()> { + let dir = src.join(rel); + for entry in std::fs::read_dir(&dir).with_context(|| format!("read {}", dir.display()))? { + let entry = entry.with_context(|| format!("read {}", dir.display()))?; + let rel = rel.join(entry.file_name()); + if is_cache(&rel) { + continue; + } + let from = src.join(&rel); + let to = dst.join(&rel); + let meta = + std::fs::symlink_metadata(&from).with_context(|| format!("stat {}", from.display()))?; + let kind = meta.file_type(); + if kind.is_dir() { + std::fs::create_dir(&to).with_context(|| format!("create {}", to.display()))?; + copy_tree(src, dst, &rel, uid, gid)?; + lchown(&to, Some(uid), Some(gid)).with_context(|| format!("chown {}", to.display()))?; + set_mode(&to, &meta)?; + } else if kind.is_symlink() { + let target = std::fs::read_link(&from) + .with_context(|| format!("read link {}", from.display()))?; + symlink(&target, &to).with_context(|| format!("link {}", to.display()))?; + lchown(&to, Some(uid), Some(gid)).with_context(|| format!("chown {}", to.display()))?; + } else if kind.is_file() { + std::fs::copy(&from, &to) + .with_context(|| format!("copy {} to {}", from.display(), to.display()))?; + if let Ok(mtime) = meta.modified() { + std::fs::File::options() + .write(true) + .open(&to) + .and_then(|f| f.set_modified(mtime)) + .with_context(|| format!("set mtime of {}", to.display()))?; + } + lchown(&to, Some(uid), Some(gid)).with_context(|| format!("chown {}", to.display()))?; + set_mode(&to, &meta)?; + } else { + tracing::info!(path = %from.display(), "salvage skips a socket, fifo or device node"); + } + } + Ok(()) +} + +/// Give `path` the permission bits of `meta`, which `chown` may have cleared. +fn set_mode(path: &Path, meta: &std::fs::Metadata) -> Result<()> { + std::fs::set_permissions(path, std::fs::Permissions::from_mode(meta.mode() & 0o7777)) + .with_context(|| format!("chmod {}", path.display())) +} + +/// The root's machine id when `etc/machine-id` is a regular file holding one. +/// The container may have replaced `etc` or the file by a link, so neither is +/// followed. +fn read_machine_id(root: &Path) -> Option { + let etc = std::fs::symlink_metadata(root.join("etc")).ok()?; + let path = root.join("etc/machine-id"); + if !etc.is_dir() || !is_regular(&path) { + return None; + } + let text = std::fs::read_to_string(path).ok()?; + let id = text.trim_end_matches('\n'); + (id.len() == 32 && id.bytes().all(|b| matches!(b, b'0'..=b'9' | b'a'..=b'f'))) + .then(|| id.to_owned()) +} + +/// Give the template a machine id when it has none: with an empty +/// `etc/machine-id` the guest makes a new one on every boot, and its journal +/// directory and DHCP identity change with it. +fn ensure_machine_id(root: &Path) -> Result<()> { + if read_machine_id(root).is_some() { + return Ok(()); + } + let uuid = std::fs::read_to_string("/proc/sys/kernel/random/uuid") + .context("read a random uuid for the machine id")?; + let id: String = uuid.trim().chars().filter(|c| *c != '-').collect(); + write_container_etc_file_in(root, "machine-id", &format!("{id}\n")) +} + +fn is_regular(path: &Path) -> bool { + std::fs::symlink_metadata(path).is_ok_and(|m| m.is_file()) +} + +/// `systemctl is-active` answers `inactive` or `failed` for a unit that is not +/// running; anything else (`active`, `activating`, `deactivating`) means the +/// root is, or is about to be, in use. +fn unit_is_stopped(container: &str) -> Result { + let out = std::process::Command::new("systemctl") + .args(["is-active", &format!("container@{container}.service")]) + .output() + .context("invoke systemctl is-active")?; + let state = String::from_utf8_lossy(&out.stdout); + Ok(matches!(state.trim(), "inactive" | "failed")) +} + +/// Today in UTC as `YYYYMMDD`. +fn today() -> String { + let secs = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |d| d.as_secs()); + date_of(secs) +} + +/// `YYYYMMDD` of a unix time, by Howard Hinnant's days-to-civil algorithm. +fn date_of(secs: u64) -> String { + let z = i64::try_from(secs / 86_400).unwrap_or(0) + 719_468; + let era = z.div_euclid(146_097); + let doe = z.rem_euclid(146_097); + let yoe = (doe - doe / 1_460 + doe / 36_524 - doe / 146_096) / 365; + let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); + let mp = (5 * doy + 2) / 153; + let day = doy - (153 * mp + 2) / 5 + 1; + let month = if mp < 10 { mp + 3 } else { mp - 9 }; + let year = yoe + era * 400 + i64::from(month <= 2); + format!("{year:04}{month:02}{day:02}") +} + +#[cfg(test)] +mod tests { + use std::os::unix::fs::{MetadataExt as _, PermissionsExt as _, lchown, symlink}; + use std::path::{Path, PathBuf}; + + use super::{MARKER, Outcome, date_of, has_content, rotate}; + use crate::shared::test_support::scratch; + + const DATE: &str = "20261028"; + + /// A stopped agent's old root and work dir under one scratch dir. The + /// "agent" is the test's own user, the one it can chown to. + struct Fixture { + top: PathBuf, + root: PathBuf, + work: PathBuf, + profiles: PathBuf, + } + + impl Fixture { + fn new() -> Self { + let top = scratch(); + let root = top.join("h-iris"); + let work = top.join("work"); + let profiles = top.join("profiles"); + std::fs::create_dir_all(&profiles).unwrap(); + std::fs::create_dir_all(root.join("home/iris")).unwrap(); + std::fs::create_dir_all(&work).unwrap(); + Self { + top, + root, + work, + profiles, + } + } + + fn put(&self, rel: &str, body: &str) { + let path = self.root.join(rel); + std::fs::create_dir_all(path.parent().unwrap()).unwrap(); + std::fs::write(path, body).unwrap(); + } + + fn rotate(&self) -> Outcome { + rotate(&self.root, "iris", &self.work, &self.profiles, DATE).unwrap() + } + + /// Where the container finds nix's per-user state, on the host. + fn profile(&self) -> PathBuf { + self.profiles.join("per-user/iris") + } + + fn salvage(&self) -> PathBuf { + self.work.join(format!("salvaged-rootfs-{DATE}")) + } + + fn names(dir: &Path) -> Vec { + let mut v: Vec = std::fs::read_dir(dir) + .unwrap() + .map(|e| e.unwrap().file_name().into_string().unwrap()) + .collect(); + v.sort(); + v + } + } + + impl Drop for Fixture { + fn drop(&mut self) { + std::fs::remove_dir_all(&self.top).ok(); + } + } + + /// The first call copies the agent's files and moves the old root aside; + /// the second finds a clean template and does nothing. + #[test] + fn salvages_once() { + let f = Fixture::new(); + f.put("home/iris/notes.txt", "keep me"); + f.put("etc/hostname", "iris"); + + let Outcome::Salvaged { salvage, old_root } = f.rotate() else { + panic!("the first call must salvage"); + }; + assert_eq!(salvage, f.salvage()); + assert_eq!( + std::fs::read_to_string(salvage.join("home/iris/notes.txt")).unwrap(), + "keep me" + ); + assert_eq!( + std::fs::read_to_string(old_root.join("home/iris/notes.txt")).unwrap(), + "keep me" + ); + assert!(f.root.join("etc").join(MARKER).is_file()); + assert!( + !f.root.join("home/iris").exists(), + "template holds the old home" + ); + + assert_eq!(f.rotate(), Outcome::Clean); + assert_eq!( + Fixture::names(&f.work), + [format!("salvaged-rootfs-{DATE}")], + "a second salvage dir appeared" + ); + } + + /// Content that appears in the template after the marker (a run by a + /// build without the flag) is salvaged again, into its own directory. + #[test] + fn content_after_the_marker_is_salvaged_again() { + let f = Fixture::new(); + f.put("home/iris/a", "1"); + f.rotate(); + f.put("home/iris/b", "2"); + let Outcome::Salvaged { salvage, .. } = f.rotate() else { + panic!("late content must be salvaged"); + }; + assert_eq!(salvage, f.work.join(format!("salvaged-rootfs-{DATE}-2"))); + assert!(salvage.join("home/iris/b").is_file()); + assert!( + f.salvage().join("home/iris/a").is_file(), + "first salvage kept" + ); + } + + /// The journal, `/tmp`, `/var/cache`, weston and the otel cursor stay out + /// of the salvage; a sibling of each does not. The root's own copies are + /// untouched. + #[test] + fn cache_paths_are_skipped_and_their_siblings_are_not() { + let f = Fixture::new(); + f.put("home/iris/keep", "x"); + for cache in [ + "tmp/scratch", + "var/cache/nix/x", + "var/log/journal/id/system.journal", + "var/lib/weston/state", + "var/lib/opentelemetry-collector/cursor", + ] { + f.put(cache, "cache"); + } + for real in [ + "var/log/other.log", + "var/lib/other/db", + "var/tmp/keep", + "root/.bash_history", + ] { + f.put(real, "real"); + } + let Outcome::Salvaged { salvage, old_root } = f.rotate() else { + panic!("must salvage"); + }; + for cache in [ + "tmp", + "var/cache", + "var/log/journal", + "var/lib/weston", + "var/lib/opentelemetry-collector", + ] { + assert!(!salvage.join(cache).exists(), "{cache} was copied"); + assert!(old_root.join(cache).exists(), "{cache} left the old root"); + } + for real in [ + "var/log/other.log", + "var/lib/other/db", + "var/tmp/keep", + "root/.bash_history", + "home/iris/keep", + ] { + assert!(salvage.join(real).is_file(), "{real} was not copied"); + } + } + + /// A root with only the cache paths and the start path's own files has + /// nothing to salvage: no directory in `work/`, no move. + #[test] + fn a_root_without_content_is_not_salvaged() { + let f = Fixture::new(); + f.put("var/cache/x", "c"); + f.put("etc/hyperhive-hive-facts", "A=b\n"); + f.put("etc/resolv.conf", "nameserver 1.1.1.1\n"); + assert_eq!(f.rotate(), Outcome::Clean); + assert!(f.work.read_dir().unwrap().next().is_none()); + assert!(f.root.join("var/cache/x").is_file(), "root was touched"); + assert!(f.root.join("etc").join(MARKER).is_file()); + // The same call with one real file makes the control: it salvages. + f.put("home/iris/real", "x"); + assert!(matches!( + rotate(&f.root, "iris", &f.work, &f.profiles, DATE).unwrap(), + Outcome::Salvaged { .. } + )); + } + + /// Nothing is removed: the old root keeps every file, and a failed + /// salvage leaves the root where it was. + #[test] + fn nothing_is_deleted() { + let f = Fixture::new(); + f.put("home/iris/deep/er/file", "data"); + f.put("etc/passwd", "iris"); + let Outcome::Salvaged { old_root, .. } = f.rotate() else { + panic!("must salvage"); + }; + assert_eq!( + std::fs::read_to_string(old_root.join("home/iris/deep/er/file")).unwrap(), + "data" + ); + assert_eq!( + std::fs::read_to_string(old_root.join("etc/passwd")).unwrap(), + "iris" + ); + + let g = Fixture::new(); + g.put("home/iris/file", "data"); + std::fs::remove_dir(&g.work).unwrap(); + std::fs::write(&g.work, "not a dir").unwrap(); + assert!(rotate(&g.root, "iris", &g.work, &g.profiles, DATE).is_err()); + assert_eq!( + std::fs::read_to_string(g.root.join("home/iris/file")).unwrap(), + "data" + ); + assert!(!g.root.join("etc").join(MARKER).exists()); + } + + /// A group of the test user other than `primary`, if it has one. + fn other_group(primary: u32) -> Option { + let mut groups = [0 as libc::gid_t; 64]; + // SAFETY: `groups` has room for the 64 ids it declares. + let n = unsafe { libc::getgroups(64, groups.as_mut_ptr()) }; + let n = usize::try_from(n).ok()?; + groups[..n].iter().copied().find(|g| *g != primary) + } + + /// Modes, mtimes, symlinks and the owner reach the salvage. The owner is + /// the test user, who can chown only to itself: the check pins that the + /// requested id lands on files, directories and links alike. + #[test] + fn modes_links_and_owner_are_kept() { + let f = Fixture::new(); + f.put("home/iris/run.sh", "#!/bin/sh\n"); + std::fs::set_permissions( + f.root.join("home/iris/run.sh"), + std::fs::Permissions::from_mode(0o750), + ) + .unwrap(); + std::fs::set_permissions( + f.root.join("home/iris"), + std::fs::Permissions::from_mode(0o700), + ) + .unwrap(); + symlink("run.sh", f.root.join("home/iris/link")).unwrap(); + let mtime = std::fs::metadata(f.root.join("home/iris/run.sh")) + .unwrap() + .mtime(); + // An unprivileged user can chown a file to any group it is in, so a + // second group gives the owner something to differ from the group new + // files get. Without one the gid check below cannot tell a chown from + // none. + let primary = std::fs::metadata(&f.work).unwrap().gid(); + if let Some(other) = other_group(primary) { + lchown(f.root.join("home/iris"), None, Some(other)).unwrap(); + } + + let Outcome::Salvaged { salvage, old_root } = f.rotate() else { + panic!("must salvage"); + }; + let me = std::fs::metadata(old_root.join("home/iris")).unwrap(); + let file = std::fs::symlink_metadata(salvage.join("home/iris/run.sh")).unwrap(); + assert_eq!(file.mode() & 0o7777, 0o750); + assert_eq!(file.mtime(), mtime); + assert_eq!( + std::fs::symlink_metadata(salvage.join("home/iris")) + .unwrap() + .mode() + & 0o7777, + 0o700 + ); + assert_eq!( + std::fs::read_link(salvage.join("home/iris/link")).unwrap(), + Path::new("run.sh") + ); + for rel in [ + "", + "home", + "home/iris", + "home/iris/run.sh", + "home/iris/link", + ] { + let meta = std::fs::symlink_metadata(salvage.join(rel)).unwrap(); + assert_eq!((meta.uid(), meta.gid()), (me.uid(), me.gid()), "{rel:?}"); + } + } + + /// The template keeps the old root's machine id; one with none gets a + /// fresh id that the next call keeps. + #[test] + fn the_template_has_a_stable_machine_id() { + let f = Fixture::new(); + f.put("home/iris/a", "1"); + f.put("etc/machine-id", "0123456789abcdef0123456789abcdef\n"); + f.rotate(); + let id = std::fs::read_to_string(f.root.join("etc/machine-id")).unwrap(); + assert_eq!(id, "0123456789abcdef0123456789abcdef\n"); + + let g = Fixture::new(); + g.put("etc/machine-id", ""); + g.rotate(); + let fresh = std::fs::read_to_string(g.root.join("etc/machine-id")).unwrap(); + assert_eq!(fresh.trim().len(), 32, "{fresh:?}"); + g.rotate(); + assert_eq!( + std::fs::read_to_string(g.root.join("etc/machine-id")).unwrap(), + fresh + ); + } + + /// `etc` replaced by a link to a directory holding a machine-id is not + /// followed. + #[test] + fn a_linked_etc_is_not_followed() { + let f = Fixture::new(); + let outside = f.top.join("outside"); + std::fs::create_dir_all(&outside).unwrap(); + std::fs::write( + outside.join("machine-id"), + "0123456789abcdef0123456789abcdef\n", + ) + .unwrap(); + symlink(&outside, f.root.join("etc")).unwrap(); + f.put("home/iris/a", "1"); + f.rotate(); + let id = std::fs::read_to_string(f.root.join("etc/machine-id")).unwrap(); + assert_ne!(id, "0123456789abcdef0123456789abcdef\n"); + } + + /// A root that is gone has nothing to salvage and gets a template. + #[test] + fn a_missing_root_gets_a_template() { + let f = Fixture::new(); + std::fs::remove_dir_all(&f.root).unwrap(); + assert!(!has_content(&f.root).unwrap()); + assert_eq!(f.rotate(), Outcome::Clean); + assert!(f.root.join("etc").join(MARKER).is_file()); + } + + /// What nix keeps for the agent user is copied to where the container finds + /// it next time, with its links intact; the old root keeps its own copy. A + /// directory already there is not overwritten, and a linked `.local` is not + /// followed. + #[test] + fn the_nix_profile_is_carried_over() { + let f = Fixture::new(); + f.put("home/iris/.local/state/nix/profiles/profile-1-link", "gen"); + symlink( + "profile-1-link", + f.root.join("home/iris/.local/state/nix/profiles/profile"), + ) + .unwrap(); + let Outcome::Salvaged { old_root, .. } = f.rotate() else { + panic!("must salvage"); + }; + let link = f.profile().join("profiles/profile"); + assert_eq!( + std::fs::read_link(&link).unwrap(), + Path::new("profile-1-link") + ); + assert_eq!(std::fs::read_to_string(&link).unwrap(), "gen"); + assert!( + old_root + .join("home/iris/.local/state/nix/profiles/profile-1-link") + .is_file() + ); + assert_eq!( + std::fs::metadata(f.profile()).unwrap().uid(), + std::fs::metadata(&f.work).unwrap().uid() + ); + + // An existing target stays as it is. + let g = Fixture::new(); + g.put("home/iris/.local/state/nix/profiles/profile-1-link", "old"); + std::fs::create_dir_all(g.profile()).unwrap(); + std::fs::write(g.profile().join("mine"), "new").unwrap(); + g.rotate(); + assert_eq!(Fixture::names(&g.profile()), ["mine"]); + + // An empty one, as the agent's tmpfiles rules leave it, is filled. + let e = Fixture::new(); + e.put("home/iris/.local/state/nix/profiles/profile-1-link", "old"); + std::fs::create_dir_all(e.profile()).unwrap(); + e.rotate(); + assert_eq!(Fixture::names(&e.profile()), ["profiles"]); + + // A `.local` that is a link to a directory with the same layout. + let h = Fixture::new(); + let outside = h.top.join("outside"); + std::fs::create_dir_all(outside.join("state/nix")).unwrap(); + std::fs::write(outside.join("state/nix/secret"), "host").unwrap(); + symlink(&outside, h.root.join("home/iris/.local")).unwrap(); + h.put("home/iris/other", "x"); + h.rotate(); + assert!(!h.profile().exists(), "followed a linked .local"); + } + + /// `per-user` or the agent's directory in it replaced by a link, as the + /// container can do: nothing lands where the link points, and the salvage + /// itself still completes. + #[test] + fn a_linked_profile_dir_is_not_followed() { + for linked in ["per-user", "per-user/iris"] { + let f = Fixture::new(); + f.put("home/iris/.local/state/nix/profiles/profile-1-link", "gen"); + let outside = f.top.join("outside"); + std::fs::create_dir_all(&outside).unwrap(); + if linked == "per-user/iris" { + std::fs::create_dir_all(f.profiles.join("per-user")).unwrap(); + } + symlink(&outside, f.profiles.join(linked)).unwrap(); + assert!(matches!(f.rotate(), Outcome::Salvaged { .. }), "{linked}"); + assert!( + outside.read_dir().unwrap().next().is_none(), + "{linked}: wrote through the link" + ); + assert!( + f.salvage() + .join("home/iris/.local/state/nix/profiles/profile-1-link") + .is_file() + ); + } + } + + #[test] + fn dates() { + assert_eq!(date_of(0), "19700101"); + assert_eq!(date_of(951_782_400), "20000229"); + assert_eq!(date_of(1_793_145_600), "20261028"); + } +} diff --git a/nix/agent-modules/otel.nix b/nix/agent-modules/otel.nix index 9ddbdbf8..5a622b81 100644 --- a/nix/agent-modules/otel.nix +++ b/nix/agent-modules/otel.nix @@ -222,7 +222,7 @@ in # `StateDirectory` (nixpkgs' module already sets one, `%S`, below # as `WorkingDirectory` too) — container-lifetime, not the # host-persisted state bind mount: it indexes /var/log/journal, - # itself fresh on every container recreate, so the two must share + # itself fresh on every container start, so the two must share # a lifetime or the cursor outlives the journal it points into. # `create_directory` matters here for a build-time-vs-runtime # reason, not a convenience one: `validateConfigFile` above runs diff --git a/nix/agent-modules/user.nix b/nix/agent-modules/user.nix index 79ce7ada..afd9f6dd 100644 --- a/nix/agent-modules/user.nix +++ b/nix/agent-modules/user.nix @@ -142,6 +142,26 @@ in gid = config.services.hyperhive.agent.user.gid; }; + # nix keeps a non-root user's profile under `~/.local/state/nix`, and the + # root is rebuilt at every start. Redirecting that directory into the + # per-container `/nix/var/nix/profiles` bind keeps what the agent installs + # (profile generations and the GC roots that point at them) across restarts. + # `~/.nix-profile` is the link nix makes on first use; recreating it puts + # the profile's `bin/` back on PATH before nix runs again. + systemd.tmpfiles.rules = + let + persisted = "/nix/var/nix/profiles/per-user/${userName}"; + owner = "${userName} ${userName}"; + in + [ + "d /nix/var/nix/profiles/per-user 0755 root root - -" + "d ${persisted} 0755 ${owner} - -" + "d ${homeDir}/.local 0755 ${owner} - -" + "d ${homeDir}/.local/state 0755 ${owner} - -" + "L ${homeDir}/.local/state/nix - - - - ${persisted}" + "L ${homeDir}/.nix-profile - - - - ${homeDir}/.local/state/nix/profiles/profile" + ]; + # `NOPASSWD: ALL` for the agent user. Lets claude's Bash tool # keep working with anything that expected root (systemctl, # nix-env, etc.) without prompting. Flip diff --git a/nix/module-eval/agent-user.nix b/nix/module-eval/agent-user.nix index 6c5f5af3..f6c03fff 100644 --- a/nix/module-eval/agent-user.nix +++ b/nix/module-eval/agent-user.nix @@ -25,6 +25,8 @@ let migrate = (agentWith { services.hyperhive.agent.user.name = "iris"; }) .system.activationScripts.hive-agent-user-migrate.text; + # The tmpfiles rules of the same named agent. + rules = (agentWith { services.hyperhive.agent.user.name = "iris"; }).systemd.tmpfiles.rules; cases = [ { # hive-c0re creates the host work dir as hive-core; this activation is @@ -63,6 +65,17 @@ let lib.hasInfix ''if [ -L "$socketDir" ] || { [ -e "$socketDir" ] && [ ! -d "$socketDir" ]; }; then'' migrate && lib.hasInfix ''elif [ -d "$socketDir" ]; then'' migrate; } + { + # The container root is rebuilt at every start; nix keeps a non-root + # user's profile in the home, so these rules are the only thing that puts + # it in the persistent per-container profiles bind. The `d` and `L` rules + # are each other's control: a misspelled path in one still finds the other. + name = "the agent's tmpfiles rules keep nix's per-user state in the persistent profiles bind"; + ok = + lib.elem "d /nix/var/nix/profiles/per-user/iris 0755 iris iris - -" rules + && lib.elem "L /home/iris/.local/state/nix - - - - /nix/var/nix/profiles/per-user/iris" rules + && lib.elem "L /home/iris/.nix-profile - - - - /home/iris/.local/state/nix/profiles/profile" rules; + } ]; in runGroup "agent-user" cases diff --git a/nix/vm-tests.nix b/nix/vm-tests.nix index c1e97acb..aba292c8 100644 --- a/nix/vm-tests.nix +++ b/nix/vm-tests.nix @@ -1,7 +1,8 @@ # NixOS VM tests: one derivation per test, exposed as `vmTests..`. # Kept out of `checks` because each boots VMs and needs KVM; the nightly # workflow `.forgejo/workflows/vm-nightly.yml` builds them. -{ pkgs }: +{ pkgs, hivePriv }: { + agent-ephemeral-rootfs = import ./vm-tests/agent-ephemeral-rootfs.nix { inherit pkgs hivePriv; }; hive-squid = import ./vm-tests/hive-squid.nix { inherit pkgs; }; } diff --git a/nix/vm-tests/agent-ephemeral-rootfs.nix b/nix/vm-tests/agent-ephemeral-rootfs.nix new file mode 100644 index 00000000..ed81d94b --- /dev/null +++ b/nix/vm-tests/agent-ephemeral-rootfs.nix @@ -0,0 +1,213 @@ +# `vmTests..agent-ephemeral-rootfs`: an agent container's switch from a +# persistent root to nspawn's `--ephemeral`, through the real hive-priv unit +# (nix/host-modules/hive-priv.nix, sandbox included). The container is an +# agent stand-in: the real agent user module (nix/agent-modules/user.nix) on a +# plain NixOS container, with the binds hive-c0re passes for an agent. +# +# What it pins: +# - hive-priv's `WriteNspawnFlags` salvages the persistent root before it +# passes `--ephemeral`, once, and deletes nothing; +# - after the switch a file in /home is gone at the next start while `work/` +# keeps its file and the machine id holds; +# - the guest journal is written inside the container, where the agent's +# forwarder (nix/agent-modules/otel.nix) reads it, and the host can read it; +# - a package installed into the user's nix profile is still there, and on +# PATH, after a restart. +# +# The agent's journal forwarder itself (an OTLP push to the hive's collector) is +# not booted here; the module-eval suite covers its wiring. +{ pkgs, hivePriv }: +let + agent = "iris"; + machine = "h-${agent}"; + agentDir = "/var/lib/hyperhive/agents/${agent}"; + bridge = "hive0"; + gateway = "10.42.0.1"; + + # Host side of what hive-c0re passes `WriteNspawnFlags` for an agent. + binds = map (dir: { + host_path = "${agentDir}/${dir}"; + container_path = if dir == "claude" then "/home/${agent}/.claude" else "/agents/${agent}/${dir}"; + read_only = false; + }) hostDirs; + hostDirs = [ + "state" + "work" + "harness" + "claude" + ]; + + # The options hive-priv.nix reads, without the modules that declare them. + hiveStubs = + { lib, ... }: + { + options.services.hyperhive = { + c0re.package = lib.mkOption { type = lib.types.package; }; + deploy.hive-controller.enable = lib.mkOption { + type = lib.types.bool; + default = true; + }; + swarm.forge.domain = lib.mkOption { + type = lib.types.str; + default = "forge.t.local"; + }; + }; + config.services.hyperhive.c0re.package = hivePriv; + # hive-priv.socket's group, which hive-c0re's module creates. + config.users.groups.hive-core = { }; + }; +in +pkgs.testers.runNixOSTest { + name = "agent-ephemeral-rootfs"; + + nodes.host = + { pkgs, ... }: + { + imports = [ + hiveStubs + ../host-modules/hive-priv.nix + ]; + virtualisation.memorySize = 2048; + # The agent bridge hive-priv names in the container's conf. + networking.bridges.${bridge}.interfaces = [ ]; + networking.interfaces.${bridge}.ipv4.addresses = [ + { + address = gateway; + prefixLength = 24; + } + ]; + systemd.tmpfiles.rules = map (dir: "d ${agentDir}/${dir} 0755 root root - -") hostDirs; + environment.systemPackages = [ pkgs.socat ]; + + containers.${machine} = { + autoStart = false; + privateNetwork = true; + hostBridge = bridge; + config = { + imports = [ ../agent-modules/user.nix ]; + system.stateVersion = "25.11"; + services.hyperhive.agent.user.name = agent; + # The agent's forwarder reads /var/log/journal, which journald only + # writes when it stores persistently. + services.journald.storage = "persistent"; + }; + }; + }; + + testScript = '' + import json + + MACHINE = "${machine}" + AGENT = "${agent}" + AGENT_DIR = "${agentDir}" + WORK = AGENT_DIR + "/work" + ROOT = "/var/lib/nixos-containers/" + MACHINE + BINDS = json.loads(${builtins.toJSON (builtins.toJSON binds)}) + HELLO = "${pkgs.hello}" + TREE = "${pkgs.tree}" + + def in_ct(cmd): + return host.succeed(f"nixos-container run {MACHINE} -- {cmd}").strip() + + def as_agent(script): + # A login shell: HOME and the profile's PATH are the agent's own. + return in_ct(f"runuser -l {AGENT} -c '{script}'") + + def boot(): + host.succeed(f"nixos-container start {MACHINE}") + host.wait_until_succeeds( + f"nixos-container run {MACHINE} -- systemctl is-active multi-user.target" + ) + + def stop(): + host.succeed(f"nixos-container stop {MACHINE}") + + def write_flags(): + """What hive-c0re does before every start: one request to hive-priv.""" + req = json.dumps({ + "op": "write_nspawn_flags", + "container": MACHINE, + "binds": BINDS, + "isolation": {"bridge": "${bridge}", "gateway_ip": "${gateway}"}, + }) + out = host.succeed(f"echo '{req}' | socat - UNIX-CONNECT:/run/hive/priv.sock") + assert json.loads(out.strip().splitlines()[-1])["ok"] is True, out + + def conf(): + return host.succeed(f"cat /etc/nixos-containers/{MACHINE}.conf") + + def salvage_dirs(): + return host.succeed(f"ls {WORK}").split() + + host.start() + host.wait_for_unit("multi-user.target") + host.wait_for_unit("hive-priv.socket") + + with subtest("before the switch: a persistent root"): + assert "--ephemeral" not in conf() + boot() + uid = in_ct(f"id -u {AGENT}") + as_agent("echo pre-switch > /home/${agent}/pre-switch.txt") + in_ct("sh -c 'echo scratch > /tmp/cache-file'") + # A root from before the agent module redirected nix's state: the real + # directory, holding a package. + in_ct(f"rm /home/{AGENT}/.local/state/nix") + as_agent(f"nix-env -i {HELLO}") + in_ct(f"test -d /home/{AGENT}/.local/state/nix/profiles") + stop() + host.succeed(f"test -f {ROOT}/home/{AGENT}/pre-switch.txt") + + with subtest("WriteNspawnFlags salvages the old root, then asks for --ephemeral"): + assert salvage_dirs() == [] + write_flags() + assert "--ephemeral" in conf(), conf() + dirs = salvage_dirs() + assert len(dirs) == 1 and dirs[0].startswith("salvaged-rootfs-"), dirs + salvage = f"{WORK}/{dirs[0]}" + assert host.succeed(f"cat {salvage}/home/{AGENT}/pre-switch.txt").strip() == "pre-switch" + assert host.succeed(f"stat -c %u {salvage}/home/{AGENT}/pre-switch.txt").strip() == uid + assert host.succeed(f"stat -c %u {salvage}").strip() == uid + # The cache paths stay out; the old root is kept whole. + host.fail(f"test -e {salvage}/tmp/cache-file") + old_roots = host.succeed(f"ls -d {ROOT}.pre-ephemeral-*").split() + assert len(old_roots) == 1, old_roots + host.succeed(f"test -f {old_roots[0]}/home/{AGENT}/pre-switch.txt") + host.fail(f"test -e {ROOT}/home/{AGENT}/pre-switch.txt") + # nix's state moved to the per-container profile bind, where the next + # start finds it. + host.succeed(f"test -L /nix/var/nix/profiles/per-container/{MACHINE}/per-user/{AGENT}/profiles/profile") + + with subtest("the first ephemeral start, and what survives a restart"): + boot() + machine_id = in_ct("cat /etc/machine-id") + in_ct("sh -c 'journalctl --sync; logger -t ephemeral-test journal-line'") + in_ct("sh -c 'journalctl --flush --sync'") + as_agent("echo gone > /home/${agent}/ephemeral.txt") + in_ct(f"sh -c 'echo kept > /agents/{AGENT}/work/kept.txt'") + # The journal is written in the guest, where the forwarder reads it, and + # the host reads it too. + assert in_ct("ls /var/log/journal") != "" + in_ct("journalctl -t ephemeral-test --directory=/var/log/journal | grep journal-line") + host.wait_until_succeeds(f"journalctl -M {MACHINE} -t ephemeral-test | grep journal-line") + # The package installed before the switch is still on the agent's PATH, and + # one installed now is kept with it. + as_agent("hello") + as_agent(f"nix-env -i {TREE}") + + stop() + write_flags() + boot() + assert in_ct("cat /etc/machine-id") == machine_id + in_ct(f"sh -c 'test ! -e /home/{AGENT}/ephemeral.txt'") + assert in_ct(f"cat /agents/{AGENT}/work/kept.txt") == "kept" + assert host.succeed(f"cat {WORK}/kept.txt").strip() == "kept" + as_agent("hello") + as_agent("tree --version") + + with subtest("the switch ran once"): + salvages = [d for d in salvage_dirs() if d.startswith("salvaged-rootfs-")] + assert len(salvages) == 1, salvages + host.succeed(f"ls -d {ROOT}.pre-ephemeral-* | wc -l | grep -x 1") + assert conf().count("--ephemeral") == 1, conf() + ''; +}