vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

run agent containers on an ephemeral root, salvaging the old one first

Browse source 
hive-priv copies a stopped container's root to the agent's work dir and
renames it aside before passing nspawn --ephemeral; the agent's nix state
moves to the persistent profiles bind. Adds the agent-ephemeral-rootfs vm
test and a module-eval case for the nix state redirect.
This commit is contained in:
atlas 2026-10-28 19:15:26 +01:00 • committed by mara
commit 81439fa710
16 changed files with 1238 additions and 39 deletions
Download patch file Download diff file Expand all files Collapse all files

51
docs/agent-lifecycle/persistence.md
View file

@ -341,6 +341,53 @@ Under `/var/lib/hyperhive/agents/<name>/`:
in the per-agent disk metric (`measure_agent_disk`) together with in the per-agent disk metric (`measure_agent_disk`) together with
`state/` and the container rootfs. The dashboard doesn't serve it. `state/` and the container rootfs. The dashboard doesn't serve it.
### The container root (per agent)
An agent container starts with `systemd-nspawn --ephemeral`
(`hive-priv`'s `WriteNspawnFlags` passes the flag). nspawn runs it on a
snapshot of `/var/lib/nixos-containers/h-<name>`, and the snapshot vanishes
at stop. The container loses anything the agent writes outside a bind: `/home/<name>`
apart from `.claude/`, `/etc`, `/var`, `/root`, `/tmp`. What persists is
`state/`, `work/`, `harness/`, `claude/` (at `~/.claude`), the per-container
`/nix/var/nix/profiles` and `/nix/var/nix/gcroots` binds, and the sockets dir.
`/home/<name>` holds caches and nothing the agent should keep: worktrees,
clones and builds go in `work/`.
The nix profile needs one extra step. nix keeps a non-root user's profile in
`~/.local/state/nix`, so the agent's tmpfiles rules
(`nix/agent-modules/user.nix`) point that directory into
`/nix/var/nix/profiles/per-user/<name>` (the persistent bind) and recreate
`~/.nix-profile` at every boot. Packages the agent installs with `nix-env` or
`nix profile` survive a restart, and go with the container on `destroy`.
**The one-time salvage.** The first start of an agent whose root has content
runs `hive-priv/src/rootfs.rs` before hive-priv passes `--ephemeral`, from the
`WriteNspawnFlags` that precedes every start, while the container is down:
1. Everything in the root except the cache paths (`tmp`, `var/cache`,
`var/log/journal`, `var/lib/weston`, `var/lib/opentelemetry-collector`) is
copied to `work/salvaged-rootfs-<YYYYMMDD>/`, owned by the agent, with
modes and mtimes kept. The agent finds its old `/home/<name>` there.
2. hive-priv copies the agent's nix state in the old home to the persistent profile bind, following no link in the path (the container can write there).
3. hive-priv renames the old root to `h-<name>.pre-ephemeral-<YYYYMMDD>` and
logs the path. Nothing deletes it; the operator removes it when ready.
4. A fresh template root takes its place, carrying `etc/machine-id` (the old
id, so the journal directory and DHCP identity hold) and the marker
`etc/hyperhive-ephemeral-root`.
The salvage runs again only if the template holds content outside the cache
paths, which happens when a hive without this change started the container
persistently in between; the new copy gets a numbered directory beside the
first. A failed salvage leaves the root where it was and the container starts
without the flag (hive-priv logs the error), so a failure costs the switch,
never data.
The journal still ships. The agent's collector (`nix/agent-modules/otel.nix`)
reads the guest's own `/var/log/journal` and pushes it over OTLP, so no
journal link to the host takes part; nspawn skips `--link-journal=try-guest`
in ephemeral mode without failing. The journal vanishes at stop, and with it
the lines written after the last push.
### Cross-agent access to state ### Cross-agent access to state
An agent holding the `manage_root_agent` permission gets every other An agent holding the `manage_root_agent` permission gets every other
@ -562,7 +609,9 @@ container lifetime:
`${homeDir}/.claude`.** Marker `${homeDir}/.claude`.** Marker
(`/var/lib/hive-agent-user-migrated`) guards single-shot; it's (`/var/lib/hive-agent-user-migrated`) guards single-shot; it's
only written once there's nothing left to migrate, so a `cp` only written once there's nothing left to migrate, so a `cp`
failure leaves it absent and the next boot retries. failure leaves it absent and the next boot retries. The root is
ephemeral, so the marker vanishes at every stop and the step
re-runs; with no `/root/.claude` it only writes the marker again.
`cp -an` (no-clobber) so any pre-existing files at the new `cp -an` (no-clobber) so any pre-existing files at the new
location win — never blow over data already there, and so a location win — never blow over data already there, and so a
retry after a partial copy is as safe as the first attempt. retry after a partial copy is as safe as the first attempt.

7
docs/process/gotchas.md
View file

@ -211,9 +211,10 @@ alongside the claude dir.
`/agents/<name>/work` (RW, owned by the agent user). It's the place for `/agents/<name>/work` (RW, owned by the agent user). It's the place for
git worktrees and build dirs: a claude started under `state/` walks up git worktrees and build dirs: a claude started under `state/` walks up
into `state/AGENTS.md` and takes on the agent's identity, and one into `state/AGENTS.md` and takes on the agent's identity, and one
started under `work/` doesn't. `/home/<name>` lives in the container started under `work/` doesn't. The container rootfs is ephemeral
rootfs, which `destroy` deletes; `work/` survives `destroy`/recreate (nspawn `--ephemeral`), so `/home/<name>` is empty after every restart:
and goes on `--purge`. The per-agent disk metric counts it; the anything the agent keeps goes in `work/` or `state/`. `work/` survives
`destroy`/recreate and goes on `--purge`. The per-agent disk metric counts it; the
dashboard doesn't serve it. dashboard doesn't serve it.
## Networking & ports ## Networking & ports

3
docs/scheduler/ci.md
View file

@ -119,7 +119,8 @@ builds all of them nightly at 02:30; a failing test fails that run. To run one
by hand, start the workflow from the forge UI (Actions, vm-nightly, Run by hand, start the workflow from the forge UI (Actions, vm-nightly, Run
workflow, optional `test` input), or run workflow, optional `test` input), or run
`nix build .#vmTests.x86_64-linux.<name>` on a host with KVM. Current tests: `nix build .#vmTests.x86_64-linux.<name>` on a host with KVM. Current tests:
`hive-squid`. `agent-ephemeral-rootfs` (the switch of an agent container to an ephemeral
root, through the real hive-priv unit) and `hive-squid`.
## Configuration reference ## Configuration reference

2
docs/scheduler/observability.md
View file

@ -340,7 +340,7 @@ exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
| `container.memory.usage` | `By` | gauge | `memory.current` | | `container.memory.usage` | `By` | gauge | `memory.current` |
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) | | `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) | | `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs, measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) | | `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + the root template (the live ephemeral layer isn't counted), measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) | | `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
| `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable | | `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable |
| `hyperhive.agent.closure_pullable` | `1` | gauge | `1` when this hive can fetch the agent's latest closure from the swarm's binary cache and verify its signature, else `0` with a `reason` label (`no-record`, `rev-mismatch`, `not-cached`, `no-credentials`, `fetch-failed`, `bad-signature`; see [the builder docs](../swarm/builder.md#what-hives-do-with-the-cache)). Omitted before the first probe pass, every 5 minutes | | `hyperhive.agent.closure_pullable` | `1` | gauge | `1` when this hive can fetch the agent's latest closure from the swarm's binary cache and verify its signature, else `0` with a `reason` label (`no-record`, `rev-mismatch`, `not-cached`, `no-credentials`, `fetch-failed`, `bad-signature`; see [the builder docs](../swarm/builder.md#what-hives-do-with-the-cache)). Omitted before the first probe pass, every 5 minutes |

2
docs/trust-boundary/security.md
View file

@ -261,7 +261,7 @@ known operations; there is no arbitrary command pass-through:
| `DestroyContainer` | `nixos-container destroy <name>` | | `DestroyContainer` | `nixos-container destroy <name>` |
| `ListContainers` | `nixos-container list` | | `ListContainers` | `nixos-container list` |
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) | | `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list + network isolation vars) | | `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted |
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else | | `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else |
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container | | `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy | | `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |

8
flake.nix
View file

@ -228,7 +228,13 @@
# Tests that boot a NixOS VM and need KVM. Not `checks`, so `nix flake # Tests that boot a NixOS VM and need KVM. Not `checks`, so `nix flake
# check` and the per-PR CI job never build them; # check` and the per-PR CI job never build them;
# `.forgejo/workflows/vm-nightly.yml` does. # `.forgejo/workflows/vm-nightly.yml` does.
vmTests = forAllSystems ({ pkgs, ... }: import ./nix/vm-tests.nix { inherit pkgs; }); vmTests = forAllSystems (
{ pkgs, system, ... }:
import ./nix/vm-tests.nix {
inherit pkgs;
hivePriv = self.packages.${system}.hive-priv;
}
);
checks = forAllSystems ( checks = forAllSystems (
{ {

2
hive-agent/prompts/system.md
View file

@ -33,7 +33,7 @@ Durable knowledge:
- write to `/agents/{label}/state/notes.md` (free-form) or any other path under `/agents/{label}/state/`. That directory is bind-mounted from the host and persists across container destroy/recreate — claude's `--continue` session only carries short-term context, but `/agents/{label}/state/` is forever. Read it back at the start of relevant turns to remember things across resets. - write to `/agents/{label}/state/notes.md` (free-form) or any other path under `/agents/{label}/state/`. That directory is bind-mounted from the host and persists across container destroy/recreate — claude's `--continue` session only carries short-term context, but `/agents/{label}/state/` is forever. Read it back at the start of relevant turns to remember things across resets.
**Work dir**: `/agents/{label}/work/` is the place for git worktrees, clones, build trees and other bulky scratch. It is bind-mounted from the host, persists across container destroy/recreate and is backed up. Keep `state/` for notes and small durable files. Don't keep work under `/home`, `/tmp` or other container paths — they are outside what's backed up. Start subagents with their `dir` under `work/` (that is also the default): a claude started under `state/` walks up into `state/AGENTS.md` and takes on your identity, one under `work/` doesn't. **Work dir**: `/agents/{label}/work/` is the place for git worktrees, clones, build trees and other bulky scratch. It is bind-mounted from the host, persists across container destroy/recreate and is backed up. Keep `state/` for notes and small durable files. Don't keep work under `/home`, `/tmp` or other container paths: the container discards them every time it stops. Start subagents with their `dir` under `work/` (that is also the default): a claude started under `state/` walks up into `state/AGENTS.md` and takes on your identity, one under `work/` doesn't.
Claude session (OAuth credentials) lives at `/root/.claude/` and persists across restarts. Claude session (OAuth credentials) lives at `/root/.claude/` and persists across restarts.

5
hive-priv-sock/src/lib.rs
View file

@ -330,6 +330,11 @@ pub enum PrivRequest {
/// Always writes `PRIVATE_NETWORK=1` + veth wiring: isolation is the only /// Always writes `PRIVATE_NETWORK=1` + veth wiring: isolation is the only
/// mode, so there is no request shape that yields a container sharing the /// mode, so there is no request shape that yields a container sharing the
/// host's network namespace. /// host's network namespace.
///
/// Also decides whether the container runs on an ephemeral root. When it is
/// stopped and its root holds anything beyond caches, the root is copied to
/// the agent's `work/` and renamed aside first, then `--ephemeral` is added
/// to the flags; see `hive-priv/src/rootfs.rs`.
WriteNspawnFlags { WriteNspawnFlags {
container: String, container: String,
binds: Vec<BindMount>, binds: Vec<BindMount>,

94
hive-priv/src/container_config.rs
View file

@ -27,7 +27,7 @@ pub(crate) fn handle_write_nspawn_flags(
load_credentials: &[CredentialMount], load_credentials: &[CredentialMount],
hive_facts: &HiveFacts, hive_facts: &HiveFacts,
) -> Result<(String, String)> { ) -> Result<(String, String)> {
validate_container_system_name(container)?; let agent = validate_container_system_name(container)?;
for bind in binds { for bind in binds {
validate_bind_path(&bind.host_path)?; validate_bind_path(&bind.host_path)?;
validate_bind_path(&bind.container_path)?; validate_bind_path(&bind.container_path)?;
@ -40,7 +40,24 @@ pub(crate) fn handle_write_nspawn_flags(
validate_bind_path(&cred.host_path)?; validate_bind_path(&cred.host_path)?;
} }
let hive_facts = render_hive_facts(hive_facts)?; let hive_facts = render_hive_facts(hive_facts)?;
write_nspawn_flags(container, binds, isolation, load_credentials, &hive_facts)?; // A root that cannot be salvaged keeps its persistent start: the flag is
// only ever passed for a root the salvage has dealt with.
let ephemeral = crate::rootfs::prepare(container, agent).unwrap_or_else(|e| {
tracing::error!(
%container,
error = %format!("{e:#}"),
"root salvage failed; the container starts with its persistent root"
);
false
});
write_nspawn_flags(
container,
binds,
isolation,
load_credentials,
&hive_facts,
ephemeral,
)?;
Ok((String::new(), String::new())) Ok((String::new(), String::new()))
} }
@ -267,6 +284,7 @@ fn write_nspawn_flags(
isolation: &NetworkIsolation, isolation: &NetworkIsolation,
load_credentials: &[CredentialMount], load_credentials: &[CredentialMount],
hive_facts: &str, hive_facts: &str,
ephemeral: bool,
) -> Result<()> { ) -> Result<()> {
use std::fmt::Write as _; use std::fmt::Write as _;
use std::os::unix::fs::MetadataExt as _; use std::os::unix::fs::MetadataExt as _;
@ -309,25 +327,7 @@ fn write_nspawn_flags(
out.push_str("LOCAL_ADDRESS6=\n"); out.push_str("LOCAL_ADDRESS6=\n");
let _ = writeln!(out, "HOST_BRIDGE={}", iso.bridge); let _ = writeln!(out, "HOST_BRIDGE={}", iso.bridge);
} }
let mut flags: Vec<String> = binds let flags_joined = nspawn_flags(binds, load_credentials, ephemeral).join(" ");
.iter()
.map(|b| {
let flag = if b.read_only { "--bind-ro" } else { "--bind" };
format!("{flag}={}:{}", b.host_path, b.container_path)
})
.collect();
flags.extend(git_overlay_flags(binds));
// Credential forwarding: nspawn loads each host secret into the
// container's credential store under `<name>`; inner units inherit it
// via `LoadCredential=<name>`. Validated (name charset + bind-path
// rules) in handle_write_nspawn_flags above.
for cred in load_credentials {
flags.push(format!(
"--load-credential={}:{}",
cred.name, cred.host_path
));
}
let flags_joined = flags.join(" ");
let _ = writeln!(out, "EXTRA_NSPAWN_FLAGS=\"{flags_joined}\""); let _ = writeln!(out, "EXTRA_NSPAWN_FLAGS=\"{flags_joined}\"");
// Keeps the owner and mode `nixos-container` created the file with; this // Keeps the owner and mode `nixos-container` created the file with; this
// helper only rewrites its lines. // helper only rewrites its lines.
@ -353,6 +353,38 @@ fn write_nspawn_flags(
Ok(()) Ok(())
} }
/// The flags `EXTRA_NSPAWN_FLAGS` carries, in the order nspawn needs them.
///
/// `--ephemeral` runs the container on a snapshot of its root and drops the
/// snapshot at stop; `crate::rootfs` has made the root fit for that before
/// the flag is asked for. The git overlays and credentials follow the binds
/// they refer to.
fn nspawn_flags(
binds: &[BindMount],
load_credentials: &[CredentialMount],
ephemeral: bool,
) -> Vec<String> {
let mut flags: Vec<String> = ephemeral
.then(|| "--ephemeral".to_owned())
.into_iter()
.collect();
flags.extend(binds.iter().map(|b| {
let flag = if b.read_only { "--bind-ro" } else { "--bind" };
format!("{flag}={}:{}", b.host_path, b.container_path)
}));
flags.extend(git_overlay_flags(binds));
// Credential forwarding: nspawn loads each host secret into the
// container's credential store under `<name>`; inner units inherit it
// via `LoadCredential=<name>`. Validated (name charset + bind-path
// rules) in handle_write_nspawn_flags.
flags.extend(
load_credentials
.iter()
.map(|cred| format!("--load-credential={}:{}", cred.name, cred.host_path)),
);
flags
}
/// Filename of the in-container DNS marker, in the container's own `/etc`. /// Filename of the in-container DNS marker, in the container's own `/etc`.
const BRIDGE_DNS_MARKER: &str = "hyperhive-bridge-dns"; const BRIDGE_DNS_MARKER: &str = "hyperhive-bridge-dns";
@ -360,7 +392,7 @@ const BRIDGE_DNS_MARKER: &str = "hyperhive-bridge-dns";
const HIVE_FACTS_FILE: &str = "hyperhive-hive-facts"; const HIVE_FACTS_FILE: &str = "hyperhive-hive-facts";
/// The container's `/` on the host. /// The container's `/` on the host.
fn container_rootfs(container: &str) -> PathBuf { pub(crate) fn container_rootfs(container: &str) -> PathBuf {
PathBuf::from(format!("/var/lib/nixos-containers/{container}")) PathBuf::from(format!("/var/lib/nixos-containers/{container}"))
} }
@ -469,7 +501,7 @@ fn write_bridge_dns_marker_in(rootfs: &Path, gateway_ip: &str) -> Result<()> {
/// materialises the rootfs on the first start), so this creates it first; the /// materialises the rootfs on the first start), so this creates it first; the
/// dir and the file persist through that start. Mode 0644: nothing written /// dir and the file persist through that start. Mode 0644: nothing written
/// here is a secret. /// here is a secret.
fn write_container_etc_file_in(rootfs: &Path, name: &str, contents: &str) -> Result<()> { pub(crate) fn write_container_etc_file_in(rootfs: &Path, name: &str, contents: &str) -> Result<()> {
use std::os::unix::fs::OpenOptionsExt as _; use std::os::unix::fs::OpenOptionsExt as _;
let etc_path = rootfs.join("etc"); let etc_path = rootfs.join("etc");
@ -510,7 +542,7 @@ fn write_container_etc_file_in(rootfs: &Path, name: &str, contents: &str) -> Res
mod tests { mod tests {
use super::{ use super::{
BindMount, HiveFacts, NetworkIsolation, git_overlay_flags, handle_write_nspawn_flags, BindMount, HiveFacts, NetworkIsolation, git_overlay_flags, handle_write_nspawn_flags,
limits_dropin_body, publish_file, remove_service_dropin, render_hive_facts, limits_dropin_body, nspawn_flags, publish_file, remove_service_dropin, render_hive_facts,
validate_container_system_name, write_bridge_dns_marker_in, write_resource_limits, validate_container_system_name, write_bridge_dns_marker_in, write_resource_limits,
}; };
use crate::shared::test_support::{entries, scratch}; use crate::shared::test_support::{entries, scratch};
@ -909,6 +941,20 @@ mod tests {
} }
} }
/// `--ephemeral` is asked for, not assumed: present exactly when the root has
/// been dealt with, ahead of the binds it applies to. The binds are the
/// control that the lookup finds what is there.
#[test]
fn ephemeral_is_passed_only_when_asked() {
let binds = [bind("/agents/atlas/work")];
let with = nspawn_flags(&binds, &[], true);
assert_eq!(with[0], "--ephemeral", "{with:?}");
assert!(with.iter().any(|f| f.starts_with("--bind-ro=")), "{with:?}");
let without = nspawn_flags(&binds, &[], false);
assert!(!without.iter().any(|f| f == "--ephemeral"), "{without:?}");
assert_eq!(without.len() + 1, with.len());
}
/// The swarm's forge and matrix containers are not the hive's: every /// The swarm's forge and matrix containers are not the hive's: every
/// request that names a container as a string refuses them before /// request that names a container as a string refuses them before
/// touching the filesystem. Control: an agent container name passes. /// touching the filesystem. Control: an agent container name passes.

10
hive-priv/src/main.rs
View file

@ -19,16 +19,18 @@
//! **Layout**: this file holds the socket loop and the one-arm-per-op //! **Layout**: this file holds the socket loop and the one-arm-per-op
//! dispatch table. Each op family's handlers and validation live in their //! dispatch table. Each op family's handlers and validation live in their
//! own module: [`container`] (lifecycle), [`container_config`] (nspawn flags, //! own module: [`container`] (lifecycle), [`container_config`] (nspawn flags,
//! systemd drop-ins), [`storage`] (btrfs subvolumes, quotas, snapshots, disk //! systemd drop-ins), [`rootfs`] (the salvage of a container's root before it
//! measurement), [`files`] (pause marker, socket dir, legacy tmpfiles), [`infra`] (gateway nginx, //! turns ephemeral), [`storage`] (btrfs subvolumes, quotas, snapshots, disk
//! infra containers) and [`closure`] (downloads from the swarm cache). Helpers //! measurement), [`files`] (pause marker, socket dir, legacy tmpfiles),
//! more than one family uses are in [`shared`]. //! [`infra`] (gateway nginx, infra containers) and [`closure`] (downloads from
//! the swarm cache). Helpers more than one family uses are in [`shared`].
mod closure; mod closure;
mod container; mod container;
mod container_config; mod container_config;
mod files; mod files;
mod infra; mod infra;
mod rootfs;
mod shared; mod shared;
mod storage; mod storage;

842
hive-priv/src/rootfs.rs Normal file
View file

@ -0,0 +1,842 @@
//! The agent container's non-persistent root.
//!
//! Agent containers start with `systemd-nspawn --ephemeral`: nspawn runs the
//! container on a throwaway snapshot of `/var/lib/nixos-containers/<c>`, so
//! anything written outside the binds is gone at stop. [`prepare`] runs ahead
//! of every `WriteNspawnFlags` and decides whether the flag may be passed.
//!
//! A root that holds anything beyond what the start path itself writes is
//! first salvaged. Everything outside [`CACHE`] is copied into
//! `<agent>/work/salvaged-rootfs-<YYYYMMDD>/`, owned by the agent, the old
//! root is renamed to `<root>.pre-ephemeral-<YYYYMMDD>` (never deleted), and a
//! fresh template root takes its place. The template is what nspawn snapshots
//! on every start, so it must stay small and free of the previous contents.
//! The template carries [`MARKER`]; it is written last, so a crash anywhere
//! earlier leaves the old root in place and the next call tries again.
//!
//! Only a stopped container is touched: a running one keeps whatever flag its
//! root already warrants.
use std::os::unix::fs::{MetadataExt as _, PermissionsExt as _, lchown, symlink};
use std::path::{Path, PathBuf};
use anyhow::{Context as _, Result, bail};
use hive_priv_sock::AGENT_STATE_ROOT;
use crate::container_config::{container_rootfs, write_container_etc_file_in};
/// The per-container nix profile bind on the host, which persists across starts.
/// What nix keeps for the agent user is carried here; see [`carry_nix_state`].
const PROFILES_ROOT: &str = "/nix/var/nix/profiles/per-container";
/// Written into a template root once it is ready to be snapshotted.
const MARKER: &str = "hyperhive-ephemeral-root";
/// Files in `etc/` that the start path writes into every root, the template
/// included. They are not content: the root counts as clean with only these.
const TEMPLATE_FILES: [&str; 6] = [
MARKER,
"hyperhive-bridge-dns",
"hyperhive-hive-facts",
"machine-id",
"os-release",
"resolv.conf",
];
/// Paths relative to the root that the salvage skips: the guest journal, its
/// forwarder's read cursor, weston's state and scratch space. The first start
/// of a container rebuilds all of them.
const CACHE: [&str; 5] = [
"tmp",
"var/cache",
"var/log/journal",
"var/lib/weston",
"var/lib/opentelemetry-collector",
];
/// What [`rotate`] did.
#[derive(Debug, PartialEq, Eq)]
pub(crate) enum Outcome {
/// Nothing outside [`CACHE`] was in the root; nothing was moved.
Clean,
/// The root's content is copied to `salvage`, and the root itself kept at
/// `old_root`.
Salvaged { salvage: PathBuf, old_root: PathBuf },
}
/// Whether `container` may be started with `--ephemeral`, salvaging its root
/// first when it holds content and the container is stopped.
///
/// # Errors
/// Returns an error when the unit state cannot be read or the salvage fails.
/// Nothing is deleted on that path, and the caller starts the container
/// without the flag.
pub(crate) fn prepare(container: &str, agent: &str) -> Result<bool> {
let root = container_rootfs(container);
if !unit_is_stopped(container)? {
return Ok(is_regular(&root.join("etc").join(MARKER)));
}
let work = Path::new(AGENT_STATE_ROOT).join(agent).join("work");
let profiles = Path::new(PROFILES_ROOT).join(container);
match rotate(&root, agent, &work, &profiles, &today())? {
Outcome::Clean => {}
Outcome::Salvaged { salvage, old_root } => tracing::warn!(
%container,
salvage = %salvage.display(),
old_root = %old_root.display(),
"root salvaged before the first ephemeral start; the old root is kept, remove it by hand"
),
}
Ok(true)
}
/// Salvage `root` if it holds content, then leave a marked template in its
/// place. `work` is the agent's work dir on the host, `profiles` the host side of
/// the container's `/nix/var/nix/profiles`; `date` is `YYYYMMDD`.
fn rotate(root: &Path, agent: &str, work: &Path, profiles: &Path, date: &str) -> Result<Outcome> {
let outcome = if has_content(root)? {
let (uid, gid) = agent_owner(root, agent, work)?;
let machine_id = read_machine_id(root);
let salvage = fresh_dir(work, &format!("salvaged-rootfs-{date}"))?;
lchown(&salvage, Some(uid), Some(gid))
.with_context(|| format!("chown {}", salvage.display()))?;
copy_tree(root, &salvage, Path::new(""), uid, gid)
.with_context(|| format!("copy {} to {}", root.display(), salvage.display()))?;
carry_nix_state(root, agent, profiles, uid, gid)?;
let old_root = free_sibling(root, &format!("pre-ephemeral-{date}"))?;
std::fs::rename(root, &old_root)
.with_context(|| format!("move {} to {}", root.display(), old_root.display()))?;
std::fs::create_dir(root).with_context(|| format!("create {}", root.display()))?;
if let Some(id) = machine_id {
write_container_etc_file_in(root, "machine-id", &format!("{id}\n"))?;
}
Outcome::Salvaged { salvage, old_root }
} else {
Outcome::Clean
};
ensure_machine_id(root)?;
write_container_etc_file_in(root, MARKER, "")?;
Ok(outcome)
}
/// Whether anything but directories, [`TEMPLATE_FILES`] and [`CACHE`] is under
/// `root`. A missing root has none.
fn has_content(root: &Path) -> Result<bool> {
fn walk(dir: &Path, rel: &Path) -> Result<bool> {
let entries = match std::fs::read_dir(dir) {
Ok(entries) => entries,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(false),
Err(e) => return Err(e).with_context(|| format!("read {}", dir.display())),
};
for entry in entries {
let entry = entry.with_context(|| format!("read {}", dir.display()))?;
let rel = rel.join(entry.file_name());
if is_cache(&rel) {
continue;
}
let kind = entry
.file_type()
.with_context(|| format!("stat {}", entry.path().display()))?;
let is_template = TEMPLATE_FILES
.iter()
.any(|name| rel == Path::new("etc").join(name));
if kind.is_dir() {
if walk(&entry.path(), &rel)? {
return Ok(true);
}
} else if !is_template {
return Ok(true);
}
}
Ok(false)
}
walk(root, Path::new(""))
}
fn is_cache(rel: &Path) -> bool {
CACHE.iter().any(|c| rel == Path::new(c))
}
/// The uid and gid the salvage is handed to: the owner of the agent's home in
/// the old root, else of its work dir. Never root.
fn agent_owner(root: &Path, agent: &str, work: &Path) -> Result<(u32, u32)> {
let home = real_dir(root, &["home", agent]);
for dir in [home, real_dir(work, &[])].into_iter().flatten() {
if let Ok(meta) = std::fs::symlink_metadata(&dir)
&& meta.uid() != 0
{
return Ok((meta.uid(), meta.gid()));
}
}
bail!(
"agent {agent}: neither its home in {} nor {} has a non-root owner",
root.display(),
work.display()
)
}
/// `base` joined with `parts`, if every component is a real directory. The
/// container could have made any of them a link, and this runs as root.
fn real_dir(base: &Path, parts: &[&str]) -> Option<PathBuf> {
let mut path = base.to_path_buf();
if !std::fs::symlink_metadata(&path).ok()?.is_dir() {
return None;
}
for part in parts {
path.push(part);
if !std::fs::symlink_metadata(&path).ok()?.is_dir() {
return None;
}
}
Some(path)
}
/// nix keeps a non-root user's profile under `~/.local/state/nix`, which the
/// agent's tmpfiles rules (`nix/agent-modules/user.nix`) point at
/// `per-user/<agent>` in `profiles`. A root that was lived in has the real
/// directory there: copy it across, so the agent's installed packages and the
/// GC roots that name them are still valid after the switch. A target that
/// holds anything is left alone; so is the whole copy when a component of the
/// target path is not a real directory, as the container can write `profiles`.
fn carry_nix_state(root: &Path, agent: &str, profiles: &Path, uid: u32, gid: u32) -> Result<()> {
let Some(src) = real_dir(root, &["home", agent, ".local", "state", "nix"]) else {
return Ok(());
};
let Some(target) = own_dir(profiles, &["per-user", agent], uid, gid)? else {
tracing::warn!(
profiles = %profiles.display(),
"a link or file where the nix state goes; the agent's nix state is in the salvage only"
);
return Ok(());
};
// The tmpfiles rules create `target` empty on a boot of the new module;
// anything in it is the agent's newer state.
if std::fs::read_dir(&target)
.with_context(|| format!("read {}", target.display()))?
.next()
.is_some()
{
return Ok(());
}
copy_tree(&src, &target, Path::new(""), uid, gid)
.with_context(|| format!("copy {} to {}", src.display(), target.display()))
}
/// `base` joined with `parts`, creating each missing component (the last one
/// owned by `uid:gid`), or `None` when `base` or an existing component is not a
/// real directory. The container could have replaced any of them by a link, and
/// this runs as root.
fn own_dir(base: &Path, parts: &[&str], uid: u32, gid: u32) -> Result<Option<PathBuf>> {
let mut path = base.to_path_buf();
if !std::fs::symlink_metadata(&path).is_ok_and(|m| m.is_dir()) {
return Ok(None);
}
for (i, part) in parts.iter().enumerate() {
path.push(part);
match std::fs::symlink_metadata(&path) {
Ok(meta) if meta.is_dir() => {}
Ok(_) => return Ok(None),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
std::fs::create_dir(&path).with_context(|| format!("create {}", path.display()))?;
if i + 1 == parts.len() {
lchown(&path, Some(uid), Some(gid))
.with_context(|| format!("chown {}", path.display()))?;
}
}
Err(e) => return Err(e).with_context(|| format!("stat {}", path.display())),
}
}
Ok(Some(path))
}
/// Create `parent/base`, or `parent/base-2`, ... when taken. Created, never
/// reused: `mkdir` does not follow a symlink an agent left under that name.
fn fresh_dir(parent: &Path, base: &str) -> Result<PathBuf> {
for n in 1.. {
let path = if n == 1 {
parent.join(base)
} else {
parent.join(format!("{base}-{n}"))
};
match std::fs::create_dir(&path) {
Ok(()) => return Ok(path),
Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => {}
Err(e) => return Err(e).with_context(|| format!("create {}", path.display())),
}
}
unreachable!("the loop above returns or continues")
}
/// `<root>.<suffix>`, or `<root>.<suffix>-2`, ... the first name nothing uses.
fn free_sibling(root: &Path, suffix: &str) -> Result<PathBuf> {
let name = root
.file_name()
.and_then(|n| n.to_str())
.with_context(|| format!("{} has no usable name", root.display()))?;
for n in 1.. {
let candidate = if n == 1 {
format!("{name}.{suffix}")
} else {
format!("{name}.{suffix}-{n}")
};
let path = root.with_file_name(candidate);
if std::fs::symlink_metadata(&path).is_err() {
return Ok(path);
}
}
unreachable!("the loop above returns or continues")
}
/// Copy everything under `src` except [`CACHE`] into `dst`, owned by
/// `uid:gid`, with modes and mtimes kept. Sockets, fifos and device nodes are
/// skipped; hard links become separate copies.
fn copy_tree(src: &Path, dst: &Path, rel: &Path, uid: u32, gid: u32) -> Result<()> {
let dir = src.join(rel);
for entry in std::fs::read_dir(&dir).with_context(|| format!("read {}", dir.display()))? {
let entry = entry.with_context(|| format!("read {}", dir.display()))?;
let rel = rel.join(entry.file_name());
if is_cache(&rel) {
continue;
}
let from = src.join(&rel);
let to = dst.join(&rel);
let meta =
std::fs::symlink_metadata(&from).with_context(|| format!("stat {}", from.display()))?;
let kind = meta.file_type();
if kind.is_dir() {
std::fs::create_dir(&to).with_context(|| format!("create {}", to.display()))?;
copy_tree(src, dst, &rel, uid, gid)?;
lchown(&to, Some(uid), Some(gid)).with_context(|| format!("chown {}", to.display()))?;
set_mode(&to, &meta)?;
} else if kind.is_symlink() {
let target = std::fs::read_link(&from)
.with_context(|| format!("read link {}", from.display()))?;
symlink(&target, &to).with_context(|| format!("link {}", to.display()))?;
lchown(&to, Some(uid), Some(gid)).with_context(|| format!("chown {}", to.display()))?;
} else if kind.is_file() {
std::fs::copy(&from, &to)
.with_context(|| format!("copy {} to {}", from.display(), to.display()))?;
if let Ok(mtime) = meta.modified() {
std::fs::File::options()
.write(true)
.open(&to)
.and_then(|f| f.set_modified(mtime))
.with_context(|| format!("set mtime of {}", to.display()))?;
}
lchown(&to, Some(uid), Some(gid)).with_context(|| format!("chown {}", to.display()))?;
set_mode(&to, &meta)?;
} else {
tracing::info!(path = %from.display(), "salvage skips a socket, fifo or device node");
}
}
Ok(())
}
/// Give `path` the permission bits of `meta`, which `chown` may have cleared.
fn set_mode(path: &Path, meta: &std::fs::Metadata) -> Result<()> {
std::fs::set_permissions(path, std::fs::Permissions::from_mode(meta.mode() & 0o7777))
.with_context(|| format!("chmod {}", path.display()))
}
/// The root's machine id when `etc/machine-id` is a regular file holding one.
/// The container may have replaced `etc` or the file by a link, so neither is
/// followed.
fn read_machine_id(root: &Path) -> Option<String> {
let etc = std::fs::symlink_metadata(root.join("etc")).ok()?;
let path = root.join("etc/machine-id");
if !etc.is_dir() || !is_regular(&path) {
return None;
}
let text = std::fs::read_to_string(path).ok()?;
let id = text.trim_end_matches('\n');
(id.len() == 32 && id.bytes().all(|b| matches!(b, b'0'..=b'9' | b'a'..=b'f')))
.then(|| id.to_owned())
}
/// Give the template a machine id when it has none: with an empty
/// `etc/machine-id` the guest makes a new one on every boot, and its journal
/// directory and DHCP identity change with it.
fn ensure_machine_id(root: &Path) -> Result<()> {
if read_machine_id(root).is_some() {
return Ok(());
}
let uuid = std::fs::read_to_string("/proc/sys/kernel/random/uuid")
.context("read a random uuid for the machine id")?;
let id: String = uuid.trim().chars().filter(|c| *c != '-').collect();
write_container_etc_file_in(root, "machine-id", &format!("{id}\n"))
}
fn is_regular(path: &Path) -> bool {
std::fs::symlink_metadata(path).is_ok_and(|m| m.is_file())
}
/// `systemctl is-active` answers `inactive` or `failed` for a unit that is not
/// running; anything else (`active`, `activating`, `deactivating`) means the
/// root is, or is about to be, in use.
fn unit_is_stopped(container: &str) -> Result<bool> {
let out = std::process::Command::new("systemctl")
.args(["is-active", &format!("container@{container}.service")])
.output()
.context("invoke systemctl is-active")?;
let state = String::from_utf8_lossy(&out.stdout);
Ok(matches!(state.trim(), "inactive" | "failed"))
}
/// Today in UTC as `YYYYMMDD`.
fn today() -> String {
let secs = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map_or(0, |d| d.as_secs());
date_of(secs)
}
/// `YYYYMMDD` of a unix time, by Howard Hinnant's days-to-civil algorithm.
fn date_of(secs: u64) -> String {
let z = i64::try_from(secs / 86_400).unwrap_or(0) + 719_468;
let era = z.div_euclid(146_097);
let doe = z.rem_euclid(146_097);
let yoe = (doe - doe / 1_460 + doe / 36_524 - doe / 146_096) / 365;
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
let mp = (5 * doy + 2) / 153;
let day = doy - (153 * mp + 2) / 5 + 1;
let month = if mp < 10 { mp + 3 } else { mp - 9 };
let year = yoe + era * 400 + i64::from(month <= 2);
format!("{year:04}{month:02}{day:02}")
}
#[cfg(test)]
mod tests {
use std::os::unix::fs::{MetadataExt as _, PermissionsExt as _, lchown, symlink};
use std::path::{Path, PathBuf};
use super::{MARKER, Outcome, date_of, has_content, rotate};
use crate::shared::test_support::scratch;
const DATE: &str = "20261028";
/// A stopped agent's old root and work dir under one scratch dir. The
/// "agent" is the test's own user, the one it can chown to.
struct Fixture {
top: PathBuf,
root: PathBuf,
work: PathBuf,
profiles: PathBuf,
}
impl Fixture {
fn new() -> Self {
let top = scratch();
let root = top.join("h-iris");
let work = top.join("work");
let profiles = top.join("profiles");
std::fs::create_dir_all(&profiles).unwrap();
std::fs::create_dir_all(root.join("home/iris")).unwrap();
std::fs::create_dir_all(&work).unwrap();
Self {
top,
root,
work,
profiles,
}
}
fn put(&self, rel: &str, body: &str) {
let path = self.root.join(rel);
std::fs::create_dir_all(path.parent().unwrap()).unwrap();
std::fs::write(path, body).unwrap();
}
fn rotate(&self) -> Outcome {
rotate(&self.root, "iris", &self.work, &self.profiles, DATE).unwrap()
}
/// Where the container finds nix's per-user state, on the host.
fn profile(&self) -> PathBuf {
self.profiles.join("per-user/iris")
}
fn salvage(&self) -> PathBuf {
self.work.join(format!("salvaged-rootfs-{DATE}"))
}
fn names(dir: &Path) -> Vec<String> {
let mut v: Vec<String> = std::fs::read_dir(dir)
.unwrap()
.map(|e| e.unwrap().file_name().into_string().unwrap())
.collect();
v.sort();
v
}
}
impl Drop for Fixture {
fn drop(&mut self) {
std::fs::remove_dir_all(&self.top).ok();
}
}
/// The first call copies the agent's files and moves the old root aside;
/// the second finds a clean template and does nothing.
#[test]
fn salvages_once() {
let f = Fixture::new();
f.put("home/iris/notes.txt", "keep me");
f.put("etc/hostname", "iris");
let Outcome::Salvaged { salvage, old_root } = f.rotate() else {
panic!("the first call must salvage");
};
assert_eq!(salvage, f.salvage());
assert_eq!(
std::fs::read_to_string(salvage.join("home/iris/notes.txt")).unwrap(),
"keep me"
);
assert_eq!(
std::fs::read_to_string(old_root.join("home/iris/notes.txt")).unwrap(),
"keep me"
);
assert!(f.root.join("etc").join(MARKER).is_file());
assert!(
!f.root.join("home/iris").exists(),
"template holds the old home"
);
assert_eq!(f.rotate(), Outcome::Clean);
assert_eq!(
Fixture::names(&f.work),
[format!("salvaged-rootfs-{DATE}")],
"a second salvage dir appeared"
);
}
/// Content that appears in the template after the marker (a run by a
/// build without the flag) is salvaged again, into its own directory.
#[test]
fn content_after_the_marker_is_salvaged_again() {
let f = Fixture::new();
f.put("home/iris/a", "1");
f.rotate();
f.put("home/iris/b", "2");
let Outcome::Salvaged { salvage, .. } = f.rotate() else {
panic!("late content must be salvaged");
};
assert_eq!(salvage, f.work.join(format!("salvaged-rootfs-{DATE}-2")));
assert!(salvage.join("home/iris/b").is_file());
assert!(
f.salvage().join("home/iris/a").is_file(),
"first salvage kept"
);
}
/// The journal, `/tmp`, `/var/cache`, weston and the otel cursor stay out
/// of the salvage; a sibling of each does not. The root's own copies are
/// untouched.
#[test]
fn cache_paths_are_skipped_and_their_siblings_are_not() {
let f = Fixture::new();
f.put("home/iris/keep", "x");
for cache in [
"tmp/scratch",
"var/cache/nix/x",
"var/log/journal/id/system.journal",
"var/lib/weston/state",
"var/lib/opentelemetry-collector/cursor",
] {
f.put(cache, "cache");
}
for real in [
"var/log/other.log",
"var/lib/other/db",
"var/tmp/keep",
"root/.bash_history",
] {
f.put(real, "real");
}
let Outcome::Salvaged { salvage, old_root } = f.rotate() else {
panic!("must salvage");
};
for cache in [
"tmp",
"var/cache",
"var/log/journal",
"var/lib/weston",
"var/lib/opentelemetry-collector",
] {
assert!(!salvage.join(cache).exists(), "{cache} was copied");
assert!(old_root.join(cache).exists(), "{cache} left the old root");
}
for real in [
"var/log/other.log",
"var/lib/other/db",
"var/tmp/keep",
"root/.bash_history",
"home/iris/keep",
] {
assert!(salvage.join(real).is_file(), "{real} was not copied");
}
}
/// A root with only the cache paths and the start path's own files has
/// nothing to salvage: no directory in `work/`, no move.
#[test]
fn a_root_without_content_is_not_salvaged() {
let f = Fixture::new();
f.put("var/cache/x", "c");
f.put("etc/hyperhive-hive-facts", "A=b\n");
f.put("etc/resolv.conf", "nameserver 1.1.1.1\n");
assert_eq!(f.rotate(), Outcome::Clean);
assert!(f.work.read_dir().unwrap().next().is_none());
assert!(f.root.join("var/cache/x").is_file(), "root was touched");
assert!(f.root.join("etc").join(MARKER).is_file());
// The same call with one real file makes the control: it salvages.
f.put("home/iris/real", "x");
assert!(matches!(
rotate(&f.root, "iris", &f.work, &f.profiles, DATE).unwrap(),
Outcome::Salvaged { .. }
));
}
/// Nothing is removed: the old root keeps every file, and a failed
/// salvage leaves the root where it was.
#[test]
fn nothing_is_deleted() {
let f = Fixture::new();
f.put("home/iris/deep/er/file", "data");
f.put("etc/passwd", "iris");
let Outcome::Salvaged { old_root, .. } = f.rotate() else {
panic!("must salvage");
};
assert_eq!(
std::fs::read_to_string(old_root.join("home/iris/deep/er/file")).unwrap(),
"data"
);
assert_eq!(
std::fs::read_to_string(old_root.join("etc/passwd")).unwrap(),
"iris"
);
let g = Fixture::new();
g.put("home/iris/file", "data");
std::fs::remove_dir(&g.work).unwrap();
std::fs::write(&g.work, "not a dir").unwrap();
assert!(rotate(&g.root, "iris", &g.work, &g.profiles, DATE).is_err());
assert_eq!(
std::fs::read_to_string(g.root.join("home/iris/file")).unwrap(),
"data"
);
assert!(!g.root.join("etc").join(MARKER).exists());
}
/// A group of the test user other than `primary`, if it has one.
fn other_group(primary: u32) -> Option<u32> {
let mut groups = [0 as libc::gid_t; 64];
// SAFETY: `groups` has room for the 64 ids it declares.
let n = unsafe { libc::getgroups(64, groups.as_mut_ptr()) };
let n = usize::try_from(n).ok()?;
groups[..n].iter().copied().find(|g| *g != primary)
}
/// Modes, mtimes, symlinks and the owner reach the salvage. The owner is
/// the test user, who can chown only to itself: the check pins that the
/// requested id lands on files, directories and links alike.
#[test]
fn modes_links_and_owner_are_kept() {
let f = Fixture::new();
f.put("home/iris/run.sh", "#!/bin/sh\n");
std::fs::set_permissions(
f.root.join("home/iris/run.sh"),
std::fs::Permissions::from_mode(0o750),
)
.unwrap();
std::fs::set_permissions(
f.root.join("home/iris"),
std::fs::Permissions::from_mode(0o700),
)
.unwrap();
symlink("run.sh", f.root.join("home/iris/link")).unwrap();
let mtime = std::fs::metadata(f.root.join("home/iris/run.sh"))
.unwrap()
.mtime();
// An unprivileged user can chown a file to any group it is in, so a
// second group gives the owner something to differ from the group new
// files get. Without one the gid check below cannot tell a chown from
// none.
let primary = std::fs::metadata(&f.work).unwrap().gid();
if let Some(other) = other_group(primary) {
lchown(f.root.join("home/iris"), None, Some(other)).unwrap();
}
let Outcome::Salvaged { salvage, old_root } = f.rotate() else {
panic!("must salvage");
};
let me = std::fs::metadata(old_root.join("home/iris")).unwrap();
let file = std::fs::symlink_metadata(salvage.join("home/iris/run.sh")).unwrap();
assert_eq!(file.mode() & 0o7777, 0o750);
assert_eq!(file.mtime(), mtime);
assert_eq!(
std::fs::symlink_metadata(salvage.join("home/iris"))
.unwrap()
.mode()
& 0o7777,
0o700
);
assert_eq!(
std::fs::read_link(salvage.join("home/iris/link")).unwrap(),
Path::new("run.sh")
);
for rel in [
"",
"home",
"home/iris",
"home/iris/run.sh",
"home/iris/link",
] {
let meta = std::fs::symlink_metadata(salvage.join(rel)).unwrap();
assert_eq!((meta.uid(), meta.gid()), (me.uid(), me.gid()), "{rel:?}");
}
}
/// The template keeps the old root's machine id; one with none gets a
/// fresh id that the next call keeps.
#[test]
fn the_template_has_a_stable_machine_id() {
let f = Fixture::new();
f.put("home/iris/a", "1");
f.put("etc/machine-id", "0123456789abcdef0123456789abcdef\n");
f.rotate();
let id = std::fs::read_to_string(f.root.join("etc/machine-id")).unwrap();
assert_eq!(id, "0123456789abcdef0123456789abcdef\n");
let g = Fixture::new();
g.put("etc/machine-id", "");
g.rotate();
let fresh = std::fs::read_to_string(g.root.join("etc/machine-id")).unwrap();
assert_eq!(fresh.trim().len(), 32, "{fresh:?}");
g.rotate();
assert_eq!(
std::fs::read_to_string(g.root.join("etc/machine-id")).unwrap(),
fresh
);
}
/// `etc` replaced by a link to a directory holding a machine-id is not
/// followed.
#[test]
fn a_linked_etc_is_not_followed() {
let f = Fixture::new();
let outside = f.top.join("outside");
std::fs::create_dir_all(&outside).unwrap();
std::fs::write(
outside.join("machine-id"),
"0123456789abcdef0123456789abcdef\n",
)
.unwrap();
symlink(&outside, f.root.join("etc")).unwrap();
f.put("home/iris/a", "1");
f.rotate();
let id = std::fs::read_to_string(f.root.join("etc/machine-id")).unwrap();
assert_ne!(id, "0123456789abcdef0123456789abcdef\n");
}
/// A root that is gone has nothing to salvage and gets a template.
#[test]
fn a_missing_root_gets_a_template() {
let f = Fixture::new();
std::fs::remove_dir_all(&f.root).unwrap();
assert!(!has_content(&f.root).unwrap());
assert_eq!(f.rotate(), Outcome::Clean);
assert!(f.root.join("etc").join(MARKER).is_file());
}
/// What nix keeps for the agent user is copied to where the container finds
/// it next time, with its links intact; the old root keeps its own copy. A
/// directory already there is not overwritten, and a linked `.local` is not
/// followed.
#[test]
fn the_nix_profile_is_carried_over() {
let f = Fixture::new();
f.put("home/iris/.local/state/nix/profiles/profile-1-link", "gen");
symlink(
"profile-1-link",
f.root.join("home/iris/.local/state/nix/profiles/profile"),
)
.unwrap();
let Outcome::Salvaged { old_root, .. } = f.rotate() else {
panic!("must salvage");
};
let link = f.profile().join("profiles/profile");
assert_eq!(
std::fs::read_link(&link).unwrap(),
Path::new("profile-1-link")
);
assert_eq!(std::fs::read_to_string(&link).unwrap(), "gen");
assert!(
old_root
.join("home/iris/.local/state/nix/profiles/profile-1-link")
.is_file()
);
assert_eq!(
std::fs::metadata(f.profile()).unwrap().uid(),
std::fs::metadata(&f.work).unwrap().uid()
);
// An existing target stays as it is.
let g = Fixture::new();
g.put("home/iris/.local/state/nix/profiles/profile-1-link", "old");
std::fs::create_dir_all(g.profile()).unwrap();
std::fs::write(g.profile().join("mine"), "new").unwrap();
g.rotate();
assert_eq!(Fixture::names(&g.profile()), ["mine"]);
// An empty one, as the agent's tmpfiles rules leave it, is filled.
let e = Fixture::new();
e.put("home/iris/.local/state/nix/profiles/profile-1-link", "old");
std::fs::create_dir_all(e.profile()).unwrap();
e.rotate();
assert_eq!(Fixture::names(&e.profile()), ["profiles"]);
// A `.local` that is a link to a directory with the same layout.
let h = Fixture::new();
let outside = h.top.join("outside");
std::fs::create_dir_all(outside.join("state/nix")).unwrap();
std::fs::write(outside.join("state/nix/secret"), "host").unwrap();
symlink(&outside, h.root.join("home/iris/.local")).unwrap();
h.put("home/iris/other", "x");
h.rotate();
assert!(!h.profile().exists(), "followed a linked .local");
}
/// `per-user` or the agent's directory in it replaced by a link, as the
/// container can do: nothing lands where the link points, and the salvage
/// itself still completes.
#[test]
fn a_linked_profile_dir_is_not_followed() {
for linked in ["per-user", "per-user/iris"] {
let f = Fixture::new();
f.put("home/iris/.local/state/nix/profiles/profile-1-link", "gen");
let outside = f.top.join("outside");
std::fs::create_dir_all(&outside).unwrap();
if linked == "per-user/iris" {
std::fs::create_dir_all(f.profiles.join("per-user")).unwrap();
}
symlink(&outside, f.profiles.join(linked)).unwrap();
assert!(matches!(f.rotate(), Outcome::Salvaged { .. }), "{linked}");
assert!(
outside.read_dir().unwrap().next().is_none(),
"{linked}: wrote through the link"
);
assert!(
f.salvage()
.join("home/iris/.local/state/nix/profiles/profile-1-link")
.is_file()
);
}
}
#[test]
fn dates() {
assert_eq!(date_of(0), "19700101");
assert_eq!(date_of(951_782_400), "20000229");
assert_eq!(date_of(1_793_145_600), "20261028");
}
}

2
nix/agent-modules/otel.nix
View file

@ -222,7 +222,7 @@ in
# `StateDirectory` (nixpkgs' module already sets one, `%S`, below # `StateDirectory` (nixpkgs' module already sets one, `%S`, below
# as `WorkingDirectory` too) — container-lifetime, not the # as `WorkingDirectory` too) — container-lifetime, not the
# host-persisted state bind mount: it indexes /var/log/journal, # host-persisted state bind mount: it indexes /var/log/journal,
# itself fresh on every container recreate, so the two must share # itself fresh on every container start, so the two must share
# a lifetime or the cursor outlives the journal it points into. # a lifetime or the cursor outlives the journal it points into.
# `create_directory` matters here for a build-time-vs-runtime # `create_directory` matters here for a build-time-vs-runtime
# reason, not a convenience one: `validateConfigFile` above runs # reason, not a convenience one: `validateConfigFile` above runs

20
nix/agent-modules/user.nix
View file

@ -142,6 +142,26 @@ in
gid = config.services.hyperhive.agent.user.gid; gid = config.services.hyperhive.agent.user.gid;
}; };
# nix keeps a non-root user's profile under `~/.local/state/nix`, and the
# root is rebuilt at every start. Redirecting that directory into the
# per-container `/nix/var/nix/profiles` bind keeps what the agent installs
# (profile generations and the GC roots that point at them) across restarts.
# `~/.nix-profile` is the link nix makes on first use; recreating it puts
# the profile's `bin/` back on PATH before nix runs again.
systemd.tmpfiles.rules =
let
persisted = "/nix/var/nix/profiles/per-user/${userName}";
owner = "${userName} ${userName}";
in
[
"d /nix/var/nix/profiles/per-user 0755 root root - -"
"d ${persisted} 0755 ${owner} - -"
"d ${homeDir}/.local 0755 ${owner} - -"
"d ${homeDir}/.local/state 0755 ${owner} - -"
"L ${homeDir}/.local/state/nix - - - - ${persisted}"
"L ${homeDir}/.nix-profile - - - - ${homeDir}/.local/state/nix/profiles/profile"
];
# `NOPASSWD: ALL` for the agent user. Lets claude's Bash tool # `NOPASSWD: ALL` for the agent user. Lets claude's Bash tool
# keep working with anything that expected root (systemctl, # keep working with anything that expected root (systemctl,
# nix-env, etc.) without prompting. Flip # nix-env, etc.) without prompting. Flip

13
nix/module-eval/agent-user.nix
View file

@ -25,6 +25,8 @@ let
migrate = migrate =
(agentWith { services.hyperhive.agent.user.name = "iris"; }) (agentWith { services.hyperhive.agent.user.name = "iris"; })
.system.activationScripts.hive-agent-user-migrate.text; .system.activationScripts.hive-agent-user-migrate.text;
# The tmpfiles rules of the same named agent.
rules = (agentWith { services.hyperhive.agent.user.name = "iris"; }).systemd.tmpfiles.rules;
cases = [ cases = [
{ {
# hive-c0re creates the host work dir as hive-core; this activation is # hive-c0re creates the host work dir as hive-core; this activation is
@ -63,6 +65,17 @@ let
lib.hasInfix ''if [ -L "$socketDir" ] || { [ -e "$socketDir" ] && [ ! -d "$socketDir" ]; }; then'' migrate lib.hasInfix ''if [ -L "$socketDir" ] || { [ -e "$socketDir" ] && [ ! -d "$socketDir" ]; }; then'' migrate
&& lib.hasInfix ''elif [ -d "$socketDir" ]; then'' migrate; && lib.hasInfix ''elif [ -d "$socketDir" ]; then'' migrate;
} }
{
# The container root is rebuilt at every start; nix keeps a non-root
# user's profile in the home, so these rules are the only thing that puts
# it in the persistent per-container profiles bind. The `d` and `L` rules
# are each other's control: a misspelled path in one still finds the other.
name = "the agent's tmpfiles rules keep nix's per-user state in the persistent profiles bind";
ok =
lib.elem "d /nix/var/nix/profiles/per-user/iris 0755 iris iris - -" rules
&& lib.elem "L /home/iris/.local/state/nix - - - - /nix/var/nix/profiles/per-user/iris" rules
&& lib.elem "L /home/iris/.nix-profile - - - - /home/iris/.local/state/nix/profiles/profile" rules;
}
]; ];
in in
runGroup "agent-user" cases runGroup "agent-user" cases

3
nix/vm-tests.nix
View file

@ -1,7 +1,8 @@
# NixOS VM tests: one derivation per test, exposed as `vmTests.<system>.<name>`. # NixOS VM tests: one derivation per test, exposed as `vmTests.<system>.<name>`.
# Kept out of `checks` because each boots VMs and needs KVM; the nightly # Kept out of `checks` because each boots VMs and needs KVM; the nightly
# workflow `.forgejo/workflows/vm-nightly.yml` builds them. # workflow `.forgejo/workflows/vm-nightly.yml` builds them.
{ pkgs }: { pkgs, hivePriv }:
{ {
agent-ephemeral-rootfs = import ./vm-tests/agent-ephemeral-rootfs.nix { inherit pkgs hivePriv; };
hive-squid = import ./vm-tests/hive-squid.nix { inherit pkgs; }; hive-squid = import ./vm-tests/hive-squid.nix { inherit pkgs; };
} }

213
nix/vm-tests/agent-ephemeral-rootfs.nix Normal file
View file

@ -0,0 +1,213 @@
# `vmTests.<system>.agent-ephemeral-rootfs`: an agent container's switch from a
# persistent root to nspawn's `--ephemeral`, through the real hive-priv unit
# (nix/host-modules/hive-priv.nix, sandbox included). The container is an
# agent stand-in: the real agent user module (nix/agent-modules/user.nix) on a
# plain NixOS container, with the binds hive-c0re passes for an agent.
#
# What it pins:
# - hive-priv's `WriteNspawnFlags` salvages the persistent root before it
# passes `--ephemeral`, once, and deletes nothing;
# - after the switch a file in /home is gone at the next start while `work/`
# keeps its file and the machine id holds;
# - the guest journal is written inside the container, where the agent's
# forwarder (nix/agent-modules/otel.nix) reads it, and the host can read it;
# - a package installed into the user's nix profile is still there, and on
# PATH, after a restart.
#
# The agent's journal forwarder itself (an OTLP push to the hive's collector) is
# not booted here; the module-eval suite covers its wiring.
{ pkgs, hivePriv }:
let
agent = "iris";
machine = "h-${agent}";
agentDir = "/var/lib/hyperhive/agents/${agent}";
bridge = "hive0";
gateway = "10.42.0.1";
# Host side of what hive-c0re passes `WriteNspawnFlags` for an agent.
binds = map (dir: {
host_path = "${agentDir}/${dir}";
container_path = if dir == "claude" then "/home/${agent}/.claude" else "/agents/${agent}/${dir}";
read_only = false;
}) hostDirs;
hostDirs = [
"state"
"work"
"harness"
"claude"
];
# The options hive-priv.nix reads, without the modules that declare them.
hiveStubs =
{ lib, ... }:
{
options.services.hyperhive = {
c0re.package = lib.mkOption { type = lib.types.package; };
deploy.hive-controller.enable = lib.mkOption {
type = lib.types.bool;
default = true;
};
swarm.forge.domain = lib.mkOption {
type = lib.types.str;
default = "forge.t.local";
};
};
config.services.hyperhive.c0re.package = hivePriv;
# hive-priv.socket's group, which hive-c0re's module creates.
config.users.groups.hive-core = { };
};
in
pkgs.testers.runNixOSTest {
name = "agent-ephemeral-rootfs";
nodes.host =
{ pkgs, ... }:
{
imports = [
hiveStubs
../host-modules/hive-priv.nix
];
virtualisation.memorySize = 2048;
# The agent bridge hive-priv names in the container's conf.
networking.bridges.${bridge}.interfaces = [ ];
networking.interfaces.${bridge}.ipv4.addresses = [
{
address = gateway;
prefixLength = 24;
}
];
systemd.tmpfiles.rules = map (dir: "d ${agentDir}/${dir} 0755 root root - -") hostDirs;
environment.systemPackages = [ pkgs.socat ];
containers.${machine} = {
autoStart = false;
privateNetwork = true;
hostBridge = bridge;
config = {
imports = [ ../agent-modules/user.nix ];
system.stateVersion = "25.11";
services.hyperhive.agent.user.name = agent;
# The agent's forwarder reads /var/log/journal, which journald only
# writes when it stores persistently.
services.journald.storage = "persistent";
};
};
};
testScript = ''
import json
MACHINE = "${machine}"
AGENT = "${agent}"
AGENT_DIR = "${agentDir}"
WORK = AGENT_DIR + "/work"
ROOT = "/var/lib/nixos-containers/" + MACHINE
BINDS = json.loads(${builtins.toJSON (builtins.toJSON binds)})
HELLO = "${pkgs.hello}"
TREE = "${pkgs.tree}"
def in_ct(cmd):
return host.succeed(f"nixos-container run {MACHINE} -- {cmd}").strip()
def as_agent(script):
# A login shell: HOME and the profile's PATH are the agent's own.
return in_ct(f"runuser -l {AGENT} -c '{script}'")
def boot():
host.succeed(f"nixos-container start {MACHINE}")
host.wait_until_succeeds(
f"nixos-container run {MACHINE} -- systemctl is-active multi-user.target"
)
def stop():
host.succeed(f"nixos-container stop {MACHINE}")
def write_flags():
"""What hive-c0re does before every start: one request to hive-priv."""
req = json.dumps({
"op": "write_nspawn_flags",
"container": MACHINE,
"binds": BINDS,
"isolation": {"bridge": "${bridge}", "gateway_ip": "${gateway}"},
})
out = host.succeed(f"echo '{req}' | socat - UNIX-CONNECT:/run/hive/priv.sock")
assert json.loads(out.strip().splitlines()[-1])["ok"] is True, out
def conf():
return host.succeed(f"cat /etc/nixos-containers/{MACHINE}.conf")
def salvage_dirs():
return host.succeed(f"ls {WORK}").split()
host.start()
host.wait_for_unit("multi-user.target")
host.wait_for_unit("hive-priv.socket")
with subtest("before the switch: a persistent root"):
assert "--ephemeral" not in conf()
boot()
uid = in_ct(f"id -u {AGENT}")
as_agent("echo pre-switch > /home/${agent}/pre-switch.txt")
in_ct("sh -c 'echo scratch > /tmp/cache-file'")
# A root from before the agent module redirected nix's state: the real
# directory, holding a package.
in_ct(f"rm /home/{AGENT}/.local/state/nix")
as_agent(f"nix-env -i {HELLO}")
in_ct(f"test -d /home/{AGENT}/.local/state/nix/profiles")
stop()
host.succeed(f"test -f {ROOT}/home/{AGENT}/pre-switch.txt")
with subtest("WriteNspawnFlags salvages the old root, then asks for --ephemeral"):
assert salvage_dirs() == []
write_flags()
assert "--ephemeral" in conf(), conf()
dirs = salvage_dirs()
assert len(dirs) == 1 and dirs[0].startswith("salvaged-rootfs-"), dirs
salvage = f"{WORK}/{dirs[0]}"
assert host.succeed(f"cat {salvage}/home/{AGENT}/pre-switch.txt").strip() == "pre-switch"
assert host.succeed(f"stat -c %u {salvage}/home/{AGENT}/pre-switch.txt").strip() == uid
assert host.succeed(f"stat -c %u {salvage}").strip() == uid
# The cache paths stay out; the old root is kept whole.
host.fail(f"test -e {salvage}/tmp/cache-file")
old_roots = host.succeed(f"ls -d {ROOT}.pre-ephemeral-*").split()
assert len(old_roots) == 1, old_roots
host.succeed(f"test -f {old_roots[0]}/home/{AGENT}/pre-switch.txt")
host.fail(f"test -e {ROOT}/home/{AGENT}/pre-switch.txt")
# nix's state moved to the per-container profile bind, where the next
# start finds it.
host.succeed(f"test -L /nix/var/nix/profiles/per-container/{MACHINE}/per-user/{AGENT}/profiles/profile")
with subtest("the first ephemeral start, and what survives a restart"):
boot()
machine_id = in_ct("cat /etc/machine-id")
in_ct("sh -c 'journalctl --sync; logger -t ephemeral-test journal-line'")
in_ct("sh -c 'journalctl --flush --sync'")
as_agent("echo gone > /home/${agent}/ephemeral.txt")
in_ct(f"sh -c 'echo kept > /agents/{AGENT}/work/kept.txt'")
# The journal is written in the guest, where the forwarder reads it, and
# the host reads it too.
assert in_ct("ls /var/log/journal") != ""
in_ct("journalctl -t ephemeral-test --directory=/var/log/journal | grep journal-line")
host.wait_until_succeeds(f"journalctl -M {MACHINE} -t ephemeral-test | grep journal-line")
# The package installed before the switch is still on the agent's PATH, and
# one installed now is kept with it.
as_agent("hello")
as_agent(f"nix-env -i {TREE}")
stop()
write_flags()
boot()
assert in_ct("cat /etc/machine-id") == machine_id
in_ct(f"sh -c 'test ! -e /home/{AGENT}/ephemeral.txt'")
assert in_ct(f"cat /agents/{AGENT}/work/kept.txt") == "kept"
assert host.succeed(f"cat {WORK}/kept.txt").strip() == "kept"
as_agent("hello")
as_agent("tree --version")
with subtest("the switch ran once"):
salvages = [d for d in salvage_dirs() if d.startswith("salvaged-rootfs-")]
assert len(salvages) == 1, salvages
host.succeed(f"ls -d {ROOT}.pre-ephemeral-* | wc -l | grep -x 1")
assert conf().count("--ephemeral") == 1, conf()
'';
}