feat(#2862): push a snapshot to a peer hive's store over the mesh

Adds the caller the fd-passing machinery existed for: hivectl agent
<name> subvol snapshot push --peer <hive> resolves the peer, connects
to its snapshot store, writes the agent header, and hands the connected
socket to hive-priv, which runs btrfs send straight into it.

The split keeps the root helper ignorant. Everything that involves
knowing where a peer is, what the wire protocol looks like, and which
hive to trust happens in the unprivileged daemon; hive-priv only ever
receives an already-open descriptor. Once btrfs send starts, neither
process is in the data path, so a multi-gigabyte transfer costs no
per-byte work and survives a hive-c0re restart.

call_with_fd takes the descriptor by value and closes it as soon as the
kernel has it. A socket stays open until every copy closes, so holding
one back would leave the receiver waiting for an EOF that never comes:
btrfs receive blocks and this side reports success for a transfer the
peer never committed. Ownership makes that unrepresentable.

The peer's store port is a new swarm.peers.<domain>.snapshotStorePort
option rather than a constant matching the module default. A pushing
hive cannot read the receiver's configuration, so assuming 51821 would
push at a port nobody promised to listen on; absent, the push fails
naming the option. swarm_peers parses the mesh address the host module
has always rendered into HYPERHIVE_PEERS but nothing read.
This commit is contained in:
atlas 2026-07-31 21:40:34 +02:00 committed by mara
commit 282bbc3709
10 changed files with 546 additions and 9 deletions

View file

@ -11,7 +11,9 @@ use hive_priv_sock::{
BindMount, CredentialMount, InfraAction, InfraContainer, JournalQuery, NetworkIsolation,
PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream,
};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use std::os::fd::{AsRawFd as _, OwnedFd, RawFd};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader, Interest};
use tokio::net::UnixStream;
/// Send a single request to `hive-priv` and return the response.
@ -40,6 +42,140 @@ pub async fn call(req: &PrivRequest) -> Result<PrivResponse> {
}
}
/// Ancillary-data buffer sized and aligned for one `SCM_RIGHTS`
/// message. `CMSG_SPACE` is not a `const fn`, so the size is a literal
/// with room to spare; the union member supplies the `cmsghdr`
/// alignment `CMSG_FIRSTHDR` requires.
#[repr(C)]
union CmsgSpace {
_align: libc::cmsghdr,
bytes: [u8; 32],
}
/// `sendmsg` `bytes` with `fd` attached as `SCM_RIGHTS`, returning how
/// many bytes were accepted.
///
/// The descriptor rides on this one call — ancillary data cannot be
/// sent separately from payload — so the caller must not have written
/// any of `bytes` beforehand.
fn send_with_fd(sock: RawFd, bytes: &[u8], fd: RawFd) -> std::io::Result<usize> {
const FD_SIZE: usize = std::mem::size_of::<RawFd>();
let mut iov = libc::iovec {
iov_base: bytes.as_ptr().cast::<libc::c_void>().cast_mut(),
iov_len: bytes.len(),
};
let mut cmsg = CmsgSpace { bytes: [0; 32] };
// SAFETY: msghdr is a plain C struct with no invalid bit patterns;
// every field we rely on is set immediately below.
let mut msg: libc::msghdr = unsafe { std::mem::zeroed() };
msg.msg_iov = &raw mut iov;
msg.msg_iovlen = 1;
msg.msg_control = std::ptr::addr_of_mut!(cmsg.bytes).cast();
// SAFETY: CMSG_SPACE is a pure size computation.
let space = unsafe { libc::CMSG_SPACE(u32::try_from(FD_SIZE).unwrap_or(4)) };
msg.msg_controllen = space as _;
// SAFETY: the control buffer is live, aligned, and long enough for
// the header CMSG_SPACE just sized.
let hdr = unsafe { libc::CMSG_FIRSTHDR(&raw const msg) };
if hdr.is_null() {
return Err(std::io::Error::other(
"control buffer too small for SCM_RIGHTS",
));
}
// SAFETY: CMSG_LEN is a pure size computation; `hdr` points into
// our own buffer, and write_unaligned tolerates its alignment.
unsafe {
let len = libc::CMSG_LEN(u32::try_from(FD_SIZE).unwrap_or(4));
std::ptr::write_unaligned(
hdr,
libc::cmsghdr {
cmsg_len: len as _,
cmsg_level: libc::SOL_SOCKET,
cmsg_type: libc::SCM_RIGHTS,
},
);
// Copied in byte-wise: the control buffer is only cmsghdr-
// aligned, so casting CMSG_DATA to a *mut RawFd would be
// unsound even where it happens to work.
std::ptr::copy_nonoverlapping(
std::ptr::from_ref(&fd).cast::<u8>(),
libc::CMSG_DATA(hdr),
FD_SIZE,
);
}
// SAFETY: `msg` points at a live iovec over `bytes` and the control
// buffer we just filled in.
let n = unsafe { libc::sendmsg(sock, &raw const msg, 0) };
if n < 0 {
return Err(std::io::Error::last_os_error());
}
Ok(usize::try_from(n).unwrap_or_default())
}
/// Send a request to `hive-priv` with an open file descriptor attached,
/// and return the response.
///
/// The helper receives the descriptor itself — not a path or an address
/// — so it can act on something it was handed without being told what
/// that thing is or how to reach it. Used for
/// [`PrivRequest::SendAgentSnapshotToFd`], where the descriptor is a
/// socket already connected to a peer hive's snapshot store.
///
/// ⚠️ Takes the descriptor by value and closes it as soon as the kernel
/// has it, *before* awaiting the response. That is not tidiness: a
/// socket stays open until every copy of it is closed, so a caller
/// holding one back would leave the receiving end waiting for an EOF
/// that never comes — `btrfs receive` blocks, and this side reports
/// success for a transfer the peer has not committed. Passing ownership
/// makes that mistake unrepresentable.
///
/// # Errors
///
/// Fails if the socket is unreachable, the descriptor cannot be
/// attached, or hive-priv answers with something other than a terminal
/// event.
pub async fn call_with_fd(req: &PrivRequest, fd: OwnedFd) -> Result<PrivResponse> {
let mut stream = UnixStream::connect(PRIV_SOCK)
.await
.context("connect to hive-priv socket (fd-passing)")?;
let line = serde_json::to_string(req).context("serialise PrivRequest")? + "\n";
let bytes = line.as_bytes();
let sock = stream.as_raw_fd();
let raw_fd = fd.as_raw_fd();
let sent = stream
.async_io(Interest::WRITABLE, || send_with_fd(sock, bytes, raw_fd))
.await
.context("send request + descriptor to hive-priv")?;
// The kernel has duplicated the descriptor into hive-priv's queue,
// so our copy has done its job. Close it now, before waiting on the
// response: see the EOF note on this function.
drop(fd);
// A short sendmsg is legal; the descriptor went with the first
// call, so the tail is an ordinary write.
if sent < bytes.len() {
stream
.write_all(&bytes[sent..])
.await
.context("send remainder of request to hive-priv")?;
}
stream.shutdown().await.context("shutdown write half")?;
let mut resp_line = String::new();
BufReader::new(stream)
.read_line(&mut resp_line)
.await
.context("read response from hive-priv")?;
match serde_json::from_str::<PrivEvent>(&resp_line).context("parse PrivResponse")? {
PrivEvent::Done(resp) => Ok(resp),
PrivEvent::Line(_) => bail!("unexpected stream line from non-streaming priv op"),
}
}
/// Send a streaming request to `hive-priv`, calling `on_line` for each
/// `PrivEvent::Line` as it arrives, then returning the terminal
/// `PrivResponse`. Used for long-running ops (`create` / `update`).