vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

hive-priv: create agent socket dirs on start; drop hyperhive-agents.conf

Browse source 
/etc/tmpfiles.d/hyperhive-agents.conf was a boot-time backstop (#2290)
that pre-created every agent's bind sources. The start preamble already
creates them for every c0re-driven start, and on this host only hive-c0re
starts agent containers. The file was also the reason the socket dir's
owner had to be declared there, which is how it spent its life at
`0777 root root` whenever the uid could not be resolved (#4742).

- hive-priv gains `EnsureAgentSocketDir { name }`, called from
  `set_nspawn_flags` in every start path. It creates
  `/run/hive-agent/<name>` `0751 root:root` with mkdirat relative to an
  O_DIRECTORY|O_NOFOLLOW fd for the parent. An existing entry has to be a
  directory (fstatat AT_SYMLINK_NOFOLLOW); anything else is refused, and a
  directory is left alone. hive-c0re's own create_dir_all went: its /run
  is read-only under ProtectSystem=strict.
- The container's `hive-agent-user-migrate` activation chowns that dir to
  the agent user and sets 0751, the same way it already handles state/ and
  harness/. It refuses a symlink or non-directory there, since `test -d`
  and chmod follow links. No host-side passwd parse, and no window where
  the dir is world-writable.
- `/run/hyperhive/agents/<name>` stays created by hive-c0re itself
  (`ensure_agent_runtime_dir`). It holds the `mcp.sock` that hive-c0re
  binds as hive-core, so it must not become root- or agent-owned.
- The `/run/hive-agent` parent is declared in hive-priv.nix, `0755
  root:root`, instead of hive-gateway's hive-core rule. hive-priv is its
  only writer now, and hive-priv's ReadWritePaths needs it to exist.
- The manager start in `ensure_root_agent` now goes through
  `converge_start_preamble` + `start_with_fallback`. It was a bare start,
  so after a reboot the manager's bind sources existed only because of the
  tmpfiles file, and its limits drop-in did not exist at all.
- Removed: `sync_tmpfiles`, `agent_uid_gid` / `parse_passwd_uid_gid`,
  `priv_client::sync_agent_tmpfiles`, `AgentTmpfilesEntry`, the tmpfiles
  body builder and their tests, plus the three call sites.
- Legacy: hive-priv unlinks the file at every start, ignoring ENOENT.
  `SyncAgentTmpfiles` stays one release as a payload-ignoring variant that
  does the same unlink and returns Ok, for an older hive-c0re.

Salvaged from #4752: the boundary.md correction that nginx only dials,
because ProtectSystem=strict makes its /run read-only.

Behaviour change: a manual `nixos-container start h-<name>` right after a
reboot, before hive-c0re has started that agent, now fails on a missing
bind source instead of starting.

Closes #4742
This commit is contained in:
atlas 2026-09-27 04:51:07 +02:00 • committed by mara
commit 2252c55df8
21 changed files with 369 additions and 421 deletions
Download patch file Download diff file Expand all files Collapse all files

379
hive-priv/src/main.rs
View file

@ -22,10 +22,10 @@ use std::path::{Path, PathBuf};
use anyhow::{Context as _, Result, anyhow, bail};
use hive_priv_sock::{
AGENT_PREFIX, AGENT_RUNTIME_ROOT, AGENT_STATE_ROOT, AgentTmpfilesEntry, BindMount,
CredentialMount, InfraAction, InfraContainer, JournalQuery, META_DIR, MIGRATE_STAGING_ROOT,
NetworkIsolation, PAUSED_MARKER_FILE, PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse,
PrivStream, PrivStreamLine, SIBLING_CONTAINERS,
AGENT_PREFIX, AGENT_RUNTIME_ROOT, AGENT_STATE_ROOT, BindMount, CredentialMount, InfraAction,
InfraContainer, JournalQuery, META_DIR, MIGRATE_STAGING_ROOT, NetworkIsolation,
PAUSED_MARKER_FILE, PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream,
PrivStreamLine, SIBLING_CONTAINERS,
};
use serde::Serialize;
use tokio::io::{AsyncWriteExt, BufReader};
@ -49,6 +49,9 @@ async fn main() -> Result<()> {
.init();
let listener = socket_listener()?;
if let Err(e) = remove_legacy_tmpfiles() {
tracing::warn!(error = %format!("{e:#}"), "legacy tmpfiles cleanup failed");
}
tracing::info!("hive-priv listening");
loop {
@ -382,6 +385,8 @@ async fn exec(
ref load_credentials,
} => handle_write_nspawn_flags(container, binds, isolation, load_credentials),
PrivRequest::EnsureAgentSocketDir { ref name } => ensure_agent_socket_dir(name),
PrivRequest::WriteResourceLimits {
ref container,
ref memory_max,
@ -499,7 +504,10 @@ async fn exec(
.await
}
PrivRequest::SyncAgentTmpfiles { ref agents } => sync_agent_tmpfiles(agents).await,
PrivRequest::SyncAgentTmpfiles => {
remove_legacy_tmpfiles()?;
Ok((String::new(), String::new()))
}
}
}
@ -1147,9 +1155,9 @@ fn remove_service_dropin(container: &str) -> Result<(String, String)> {
/// The condition causes systemd to *skip* (not *fail*) the unit when the
/// bind-mount source dir is absent — result is `condition`, which does not
/// increment the start-limit counter. This is belt-and-braces on top of
/// the tmpfiles.d entries written by `SyncAgentTmpfiles`: in the unlikely
/// event the dir is missing at start time, the unit idles rather than
/// restart-looping into `start-limit-hit`.
/// hive-c0re creating the dir in its start preamble: if it is missing at
/// start time, the unit idles rather than restart-looping into
/// `start-limit-hit`.
fn write_resource_limits(
container: &str,
memory_max: &str,
@ -3310,126 +3318,109 @@ fn write_bridge_dns_marker_in(rootfs: &Path, gateway_ip: &str) -> Result<()> {
.publish()
}
/// `SyncAgentTmpfiles` — write `/etc/tmpfiles.d/hyperhive-agents.conf` for
/// the given agent set and immediately apply it with `systemd-tmpfiles --create`.
///
/// Each call atomically replaces the file with entries for all current agents,
/// then creates any missing dirs on the running host. The file survives reboots
/// and is read by `systemd-tmpfiles-setup.service` (runs in `sysinit.target`,
/// before any container units can start), so bind-mount source dirs are always
/// pre-created regardless of whether hive-c0re has reached `ensure_agent_runtime_dir`.
///
/// Directories written per agent:
/// - `/run/hyperhive/agents/<name>` (MCP socket dir, bind-mounted into container
/// as `/run/hive`)
/// - `/run/hive-agent/<name>` (web socket dir, bind-mounted into container)
const TMPFILES_PATH: &str = "/etc/tmpfiles.d/hyperhive-agents.conf";
/// Written by older hive-priv builds. Nothing writes it now, but left on disk
/// systemd-tmpfiles would still apply it at every boot.
const LEGACY_TMPFILES_PATH: &str = "/etc/tmpfiles.d/hyperhive-agents.conf";
/// The `tmpfiles.d` body, built without touching the filesystem.
///
/// Split out of `sync_agent_tmpfiles` so the modes below can be asserted: the
/// caller writes the file and then shells out to `systemd-tmpfiles`, neither of
/// which a test can do, and a mode nobody can assert is a mode that drifts.
fn agent_tmpfiles_content(agents: &[AgentTmpfilesEntry]) -> String {
use std::fmt::Write as _;
let mut content =
String::from("# managed by hive-c0re — do not edit (regenerated on spawn/destroy)\n");
// Parent dirs — created with permissive mode so hive-c0re can make subdirs.
// /run/hyperhive itself is also a RuntimeDirectory of hive-c0re.service; the
// tmpfiles.d entry here ensures it exists before hive-c0re starts (boot race).
//
// 0751, not 0750: must match hive-c0re.service's own `RuntimeDirectoryMode`
// and `docs/trust-boundary/boundary.md` — the extra `--x` on `other` is what
// lets a `hive-admin`-only user (no `hive-core` membership) traverse into the
// directory to reach `host.sock`; without it that user gets a permission
// denied opening the socket despite correct group membership on the socket
// itself. A mismatch here isn't just cosmetic: this line is regenerated and
// re-applied on every agent spawn/destroy via `systemd-tmpfiles --create`,
// so a stale `0750` here actively re-asserts the old, wrong mode far more
// often than a reboot does.
content.push_str("d /run/hyperhive 0751 hive-core hive-core -\n");
writeln!(content, "d {AGENT_RUNTIME_ROOT} 0755 hive-core hive-core -").ok();
// `hive-core`, not root: c0re does the `create_dir_all` for a new agent's
// subdir itself, so a root-owned parent EACCESes on the first spawn of a
// fresh host. This must stay in step with the identical rule in
// `nix/host-modules/hive-gateway/default.nix` — the two files declared
// different owners for this one path, and which won depended on the order
// systemd happened to read them in.
writeln!(content, "d {SOCKET_DIR_ROOT} 0755 hive-core hive-core -").ok();
// Per-agent dirs.
for entry in agents {
let name = &entry.name;
writeln!(
content,
"d {AGENT_RUNTIME_ROOT}/{name} 0755 hive-core hive-core -"
)
.ok();
// The agent's socket dir. Three principals need it and no two share a
// group, so the mode has to say so explicitly:
//
// owner = the agent user rwx binds + unlinks agent.sock/web.sock
// other = --x traverse only, no listing
//
// "other" covers hive-c0re (dials agent.sock) and the gateway's nginx
// (dials web.sock, and has all of /run/hive-agent bind-mounted in).
// Both sockets are 0666, so traversal is all they need.
//
// ⚠️ 0751 is load-bearing, not tidiness. A world-writable socket dir
// lets anything that can reach the path unlink an agent's socket and
// bind its own, receiving that agent's todos. Why directory-write
// confers that and the sticky bit does not save it:
// `docs/trust-boundary/boundary.md::the per-agent socket dir`.
//
// The owner is declared here because `d` re-applies on every sync, so
// a chown made anywhere else does not survive the next agent's spawn.
if let (Some(uid), Some(gid)) = (entry.uid, entry.gid) {
writeln!(content, "d {SOCKET_DIR_ROOT}/{name} 0751 {uid} {gid} -").ok();
} else {
// Before the container's /etc/passwd exists there is no uid to
// name, and the harness must still be able to bind. Keep the old
// permissive mode for that agent alone; the next sync (any spawn
// or destroy, or c0re restart) resolves the uid and tightens it.
tracing::info!(%name, "tmpfiles.d: agent uid unknown, deferring 0751 on socket dir");
writeln!(content, "d {SOCKET_DIR_ROOT}/{name} 0777 root root -").ok();
/// Unlink [`LEGACY_TMPFILES_PATH`]; already absent is success. Run at every
/// hive-priv start and by the legacy `SyncAgentTmpfiles` request.
fn remove_legacy_tmpfiles() -> Result<()> {
match std::fs::remove_file(LEGACY_TMPFILES_PATH) {
Ok(()) => {
tracing::info!("removed legacy {LEGACY_TMPFILES_PATH}");
Ok(())
}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(e) => Err(e).with_context(|| format!("remove {LEGACY_TMPFILES_PATH}")),
}
content
}
async fn sync_agent_tmpfiles(agents: &[AgentTmpfilesEntry]) -> Result<(String, String)> {
for entry in agents {
validate_agent_name(&entry.name)?;
/// `EnsureAgentSocketDir` — create `/run/hive-agent/<name>` if missing.
fn ensure_agent_socket_dir(name: &str) -> Result<(String, String)> {
ensure_socket_dir_in(Path::new(SOCKET_DIR_ROOT), name)?;
Ok((String::new(), String::new()))
}
/// Create `<root>/<name>` as a `0751` directory owned by the caller (root in
/// production), or accept an existing directory untouched.
/// `name` must be an agent ident; it is validated here, before any path is
/// built from it.
///
/// The result is an nspawn bind source, so a symlink here would bind a host
/// path of the planter's choosing into the container. Every step is relative
/// to an `O_DIRECTORY|O_NOFOLLOW` fd for `root`, and an existing entry is
/// checked with `AT_SYMLINK_NOFOLLOW` and refused unless it is a directory.
///
/// An existing directory keeps its owner and mode: the container's activation
/// hands it to the agent user, and re-asserting root here would undo that on
/// every start.
fn ensure_socket_dir_in(root: &Path, name: &str) -> Result<()> {
use std::os::unix::fs::OpenOptionsExt as _;
validate_agent_name(name)?;
let path = root.join(name);
let c_name = std::ffi::CString::new(name)
.with_context(|| format!("agent name {name:?} contains a NUL byte"))?;
let root_dir = std::fs::OpenOptions::new()
.read(true)
.custom_flags(libc::O_DIRECTORY | libc::O_NOFOLLOW | libc::O_CLOEXEC)
.open(root)
.with_context(|| format!("open (no-follow) {}", root.display()))?;
// SAFETY: `root_dir` is an open directory fd and `c_name` NUL-terminated.
if unsafe { libc::mkdirat(root_dir.as_raw_fd(), c_name.as_ptr(), 0o751) } == 0 {
// mkdirat's mode is masked by the umask; the mode is part of the
// contract, so it is set explicitly.
// SAFETY: as above.
let rc = unsafe {
libc::fchmodat(
root_dir.as_raw_fd(),
c_name.as_ptr(),
0o751,
libc::AT_SYMLINK_NOFOLLOW,
)
};
if rc != 0 {
return Err(std::io::Error::last_os_error())
.with_context(|| format!("chmod 0751 {}", path.display()));
}
tracing::info!(path = %path.display(), "created agent socket dir");
return Ok(());
}
let content = agent_tmpfiles_content(agents);
// Overlapping syncs each stage their own temp, so the last rename wins
// with one complete roster rather than a mix of two.
publish_file(Path::new(TMPFILES_PATH), content.as_bytes(), 0o644, None)?;
tracing::info!(agents = agents.len(), "tmpfiles.d: wrote {TMPFILES_PATH}");
// Apply immediately so dirs exist on the running host, not just after next boot.
let out = Command::new("systemd-tmpfiles")
.args(["--create", TMPFILES_PATH])
.output()
.await
.context("systemd-tmpfiles --create")?;
if !out.status.success() {
let stderr = String::from_utf8_lossy(&out.stderr).trim().to_owned();
anyhow::bail!(
"systemd-tmpfiles --create failed ({}): {stderr}",
out.status
let err = std::io::Error::last_os_error();
if err.kind() != std::io::ErrorKind::AlreadyExists {
return Err(err).with_context(|| format!("create {}", path.display()));
}
// SAFETY: `stat` is plain old data; the zeroed value is only read after
// `fstatat` has filled it in.
let mut st: libc::stat = unsafe { std::mem::zeroed() };
// SAFETY: as for `mkdirat`; `st` is a valid, writable `stat`.
let rc = unsafe {
libc::fstatat(
root_dir.as_raw_fd(),
c_name.as_ptr(),
&raw mut st,
libc::AT_SYMLINK_NOFOLLOW,
)
};
if rc != 0 {
return Err(std::io::Error::last_os_error())
.with_context(|| format!("stat (no-follow) {}", path.display()));
}
if st.st_mode & libc::S_IFMT != libc::S_IFDIR {
bail!(
"{} exists but is not a directory; refusing it as a bind source",
path.display()
);
}
Ok((String::new(), String::new()))
Ok(())
}
#[cfg(test)]
mod tests {
use super::{
AgentTmpfilesEntry, BindMount, BoundedRun, OwnedFd, PAUSED_MARKER_FILE, PrivRequest,
StagedFile, agent_tmpfiles_content, check_fd_agreement, clear_runner_credentials,
contains_secret_shaped_run, describe_forge_admin, ensure_plain_filename, git_overlay_flags,
BindMount, BoundedRun, OwnedFd, PAUSED_MARKER_FILE, PrivRequest, StagedFile,
check_fd_agreement, clear_runner_credentials, contains_secret_shaped_run,
describe_forge_admin, ensure_plain_filename, ensure_socket_dir_in, git_overlay_flags,
limits_dropin_body, matrix_token_filename, open_dir, open_export_dest, partial_name,
publish_file, redact_secret_line, remove_marker_in, run_bounded, single_output_path,
toplevel_attr, validate_account_name, validate_credential_name, validate_snapshot_name,
@ -3446,49 +3437,6 @@ mod tests {
}
}
/// `/run/hyperhive` is declared twice — here and as `hive-c0re.service`'s
/// `RuntimeDirectory`/`RuntimeDirectoryMode`. When the two disagree,
/// whichever runs last wins, and the loser's mode is silently re-applied on
/// every agent spawn. `0751` is the value the socket's own `0660
/// hive-admin` gate depends on: `o=--x` is what lets an admin who is not in
/// `hive-core` traverse to it at all.
#[test]
fn hyperhive_runtime_dir_keeps_the_traversable_mode() {
let out = agent_tmpfiles_content(&[]);
assert!(
out.contains("d /run/hyperhive 0751 hive-core hive-core -\n"),
"{out}"
);
// The specific regression: 0750 denies traversal before host.sock's own
// permissions are consulted.
assert!(!out.contains("d /run/hyperhive 0750"), "{out}");
}
/// Control for the assertion above: an empty roster still emits the parent
/// dirs, and a populated one adds per-agent lines — so a `contains` check
/// over this body is reading a body that was actually built.
#[test]
fn tmpfiles_body_grows_with_the_roster() {
let empty = agent_tmpfiles_content(&[]);
let one = agent_tmpfiles_content(&[AgentTmpfilesEntry {
name: "probe".to_owned(),
uid: Some(1234),
gid: Some(1234),
}]);
assert!(one.len() > empty.len(), "empty={empty}\none={one}");
assert!(
one.contains("d /run/hive-agent/probe 0751 1234 1234 -\n"),
"{one}"
);
// Every per-agent line the builder emits, so no mode here is
// unasserted — an unpinned mode is one that drifts.
assert!(
one.contains("d /run/hyperhive/agents/probe 0755 hive-core hive-core -\n"),
"{one}"
);
assert!(!empty.contains("probe"), "{empty}");
}
/// Pins the exact attr path we hand to `nix build` — it has to match
/// `hive-c0re`'s own `lifecycle::prebuild_toplevel` construction, since
/// that step's whole point is warming the store for this later build.
@ -4228,6 +4176,121 @@ mod tests {
}
}
/// A missing socket dir is created `0751`, owned by the caller (root in
/// production), which is the mode the container's activation expects to
/// hand over and the one dialers traverse.
#[test]
fn socket_dir_created_0751_when_absent() {
use std::os::unix::fs::{MetadataExt as _, PermissionsExt as _};
let dir = scratch();
ensure_socket_dir_in(&dir, "probe").unwrap();
let meta = std::fs::symlink_metadata(dir.join("probe")).unwrap();
assert!(meta.is_dir());
assert_eq!(meta.permissions().mode() & 0o7777, 0o751);
// SAFETY: `geteuid` has no preconditions.
assert_eq!(meta.uid(), unsafe { libc::geteuid() });
std::fs::remove_dir_all(&dir).ok();
}
/// An existing directory keeps its mode and contents: by the second start
/// it belongs to the agent user, and a live socket may sit inside it.
#[test]
fn socket_dir_existing_directory_left_alone() {
use std::os::unix::fs::PermissionsExt as _;
let dir = scratch();
let sub = dir.join("probe");
std::fs::create_dir(&sub).unwrap();
std::fs::set_permissions(&sub, std::fs::Permissions::from_mode(0o700)).unwrap();
std::fs::write(sub.join("web.sock"), "").unwrap();
ensure_socket_dir_in(&dir, "probe").unwrap();
let mode = std::fs::metadata(&sub).unwrap().permissions().mode() & 0o7777;
assert_eq!(mode, 0o700, "an existing dir's mode must not be reset");
assert!(sub.join("web.sock").exists());
std::fs::remove_dir_all(&dir).ok();
}
/// A symlink at the leaf would make nspawn bind wherever it points; it is
/// refused, whether it points at a directory or nowhere.
#[test]
fn socket_dir_refuses_symlink_leaf() {
use std::os::unix::fs::PermissionsExt as _;
let dir = scratch();
let elsewhere = dir.join("elsewhere");
std::fs::create_dir(&elsewhere).unwrap();
std::fs::set_permissions(&elsewhere, std::fs::Permissions::from_mode(0o700)).unwrap();
std::os::unix::fs::symlink(&elsewhere, dir.join("probe")).unwrap();
std::os::unix::fs::symlink(dir.join("missing"), dir.join("dangling")).unwrap();
assert!(ensure_socket_dir_in(&dir, "probe").is_err());
assert!(ensure_socket_dir_in(&dir, "dangling").is_err());
let mode = std::fs::metadata(&elsewhere).unwrap().permissions().mode() & 0o7777;
assert_eq!(mode, 0o700, "the symlink target must be untouched");
assert!(!dir.join("missing").exists());
std::fs::remove_dir_all(&dir).ok();
}
/// A regular file where the dir should be is refused, not bound.
#[test]
fn socket_dir_refuses_non_directory_leaf() {
let dir = scratch();
std::fs::write(dir.join("probe"), "x").unwrap();
let err = ensure_socket_dir_in(&dir, "probe").unwrap_err();
assert!(format!("{err:#}").contains("not a directory"), "{err:#}");
std::fs::remove_dir_all(&dir).ok();
}
/// A symlinked root would place the new dir in a tree of the planter's
/// choosing; the `O_NOFOLLOW` open refuses it before anything is created.
#[test]
fn socket_dir_refuses_symlinked_root() {
let dir = scratch();
let elsewhere = dir.join("elsewhere");
std::fs::create_dir(&elsewhere).unwrap();
let root = dir.join("root");
std::os::unix::fs::symlink(&elsewhere, &root).unwrap();
assert!(ensure_socket_dir_in(&root, "probe").is_err());
assert!(!elsewhere.join("probe").exists());
std::fs::remove_dir_all(&dir).ok();
}
/// Names that are not a plain agent ident are refused by validation,
/// before anything is created. Runs against a scratch parent, so the
/// refusal cannot come from a missing `/run/hive-agent`; `.`/`..` exist
/// as directories and `Atlas` would be created, so each bad name gets
/// through only if validation is gone. Control: a plain name succeeds
/// in the same parent.
#[test]
fn socket_dir_refuses_bogus_names() {
let dir = scratch();
for bad in ["", ".", "..", "../etc", "a/b", "Atlas", "a b", "a\0b"] {
let err = ensure_socket_dir_in(&dir, bad).unwrap_err();
assert!(
format!("{err:#}").contains("invalid name"),
"agent name {bad:?} must be refused by validation, got: {err:#}"
);
}
assert_eq!(std::fs::read_dir(&dir).unwrap().count(), 0);
ensure_socket_dir_in(&dir, "atlas").unwrap();
assert!(dir.join("atlas").is_dir());
std::fs::remove_dir_all(&dir).ok();
}
/// An older hive-c0re's `SyncAgentTmpfiles` still decodes, payload and
/// all, so it gets the legacy cleanup rather than a parse error. Control:
/// an unknown op does not decode.
#[test]
fn legacy_sync_agent_tmpfiles_request_still_decodes() {
let legacy =
r#"{"op":"sync_agent_tmpfiles","agents":[{"name":"atlas","uid":1000,"gid":994}]}"#;
assert!(matches!(
serde_json::from_str::<PrivRequest>(legacy),
Ok(PrivRequest::SyncAgentTmpfiles)
));
assert!(serde_json::from_str::<PrivRequest>(r#"{"op":"sync_agent_tmpfilez"}"#).is_err());
}
/// The runner-credential clear, in all three states that matter. The
/// PRESENCE arm is the load-bearing one: an implementation that did nothing
/// at all would pass the "absent is fine" arm perfectly, and the whole point