feat(#2862): push a snapshot to a peer hive's store over the mesh
Adds the caller the fd-passing machinery existed for: hivectl agent <name> subvol snapshot push --peer <hive> resolves the peer, connects to its snapshot store, writes the agent header, and hands the connected socket to hive-priv, which runs btrfs send straight into it. The split keeps the root helper ignorant. Everything that involves knowing where a peer is, what the wire protocol looks like, and which hive to trust happens in the unprivileged daemon; hive-priv only ever receives an already-open descriptor. Once btrfs send starts, neither process is in the data path, so a multi-gigabyte transfer costs no per-byte work and survives a hive-c0re restart. call_with_fd takes the descriptor by value and closes it as soon as the kernel has it. A socket stays open until every copy closes, so holding one back would leave the receiver waiting for an EOF that never comes: btrfs receive blocks and this side reports success for a transfer the peer never committed. Ownership makes that unrepresentable. The peer's store port is a new swarm.peers.<domain>.snapshotStorePort option rather than a constant matching the module default. A pushing hive cannot read the receiver's configuration, so assuming 51821 would push at a port nobody promised to listen on; absent, the push fails naming the option. swarm_peers parses the mesh address the host module has always rendered into HYPERHIVE_PEERS but nothing read.
This commit is contained in:
parent
51f352f0ca
commit
282bbc3709
10 changed files with 546 additions and 9 deletions
|
|
@ -11,7 +11,9 @@ use hive_priv_sock::{
|
|||
BindMount, CredentialMount, InfraAction, InfraContainer, JournalQuery, NetworkIsolation,
|
||||
PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream,
|
||||
};
|
||||
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
|
||||
use std::os::fd::{AsRawFd as _, OwnedFd, RawFd};
|
||||
|
||||
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader, Interest};
|
||||
use tokio::net::UnixStream;
|
||||
|
||||
/// Send a single request to `hive-priv` and return the response.
|
||||
|
|
@ -40,6 +42,140 @@ pub async fn call(req: &PrivRequest) -> Result<PrivResponse> {
|
|||
}
|
||||
}
|
||||
|
||||
/// Ancillary-data buffer sized and aligned for one `SCM_RIGHTS`
|
||||
/// message. `CMSG_SPACE` is not a `const fn`, so the size is a literal
|
||||
/// with room to spare; the union member supplies the `cmsghdr`
|
||||
/// alignment `CMSG_FIRSTHDR` requires.
|
||||
#[repr(C)]
|
||||
union CmsgSpace {
|
||||
_align: libc::cmsghdr,
|
||||
bytes: [u8; 32],
|
||||
}
|
||||
|
||||
/// `sendmsg` `bytes` with `fd` attached as `SCM_RIGHTS`, returning how
|
||||
/// many bytes were accepted.
|
||||
///
|
||||
/// The descriptor rides on this one call — ancillary data cannot be
|
||||
/// sent separately from payload — so the caller must not have written
|
||||
/// any of `bytes` beforehand.
|
||||
fn send_with_fd(sock: RawFd, bytes: &[u8], fd: RawFd) -> std::io::Result<usize> {
|
||||
const FD_SIZE: usize = std::mem::size_of::<RawFd>();
|
||||
|
||||
let mut iov = libc::iovec {
|
||||
iov_base: bytes.as_ptr().cast::<libc::c_void>().cast_mut(),
|
||||
iov_len: bytes.len(),
|
||||
};
|
||||
let mut cmsg = CmsgSpace { bytes: [0; 32] };
|
||||
// SAFETY: msghdr is a plain C struct with no invalid bit patterns;
|
||||
// every field we rely on is set immediately below.
|
||||
let mut msg: libc::msghdr = unsafe { std::mem::zeroed() };
|
||||
msg.msg_iov = &raw mut iov;
|
||||
msg.msg_iovlen = 1;
|
||||
msg.msg_control = std::ptr::addr_of_mut!(cmsg.bytes).cast();
|
||||
// SAFETY: CMSG_SPACE is a pure size computation.
|
||||
let space = unsafe { libc::CMSG_SPACE(u32::try_from(FD_SIZE).unwrap_or(4)) };
|
||||
msg.msg_controllen = space as _;
|
||||
|
||||
// SAFETY: the control buffer is live, aligned, and long enough for
|
||||
// the header CMSG_SPACE just sized.
|
||||
let hdr = unsafe { libc::CMSG_FIRSTHDR(&raw const msg) };
|
||||
if hdr.is_null() {
|
||||
return Err(std::io::Error::other(
|
||||
"control buffer too small for SCM_RIGHTS",
|
||||
));
|
||||
}
|
||||
// SAFETY: CMSG_LEN is a pure size computation; `hdr` points into
|
||||
// our own buffer, and write_unaligned tolerates its alignment.
|
||||
unsafe {
|
||||
let len = libc::CMSG_LEN(u32::try_from(FD_SIZE).unwrap_or(4));
|
||||
std::ptr::write_unaligned(
|
||||
hdr,
|
||||
libc::cmsghdr {
|
||||
cmsg_len: len as _,
|
||||
cmsg_level: libc::SOL_SOCKET,
|
||||
cmsg_type: libc::SCM_RIGHTS,
|
||||
},
|
||||
);
|
||||
// Copied in byte-wise: the control buffer is only cmsghdr-
|
||||
// aligned, so casting CMSG_DATA to a *mut RawFd would be
|
||||
// unsound even where it happens to work.
|
||||
std::ptr::copy_nonoverlapping(
|
||||
std::ptr::from_ref(&fd).cast::<u8>(),
|
||||
libc::CMSG_DATA(hdr),
|
||||
FD_SIZE,
|
||||
);
|
||||
}
|
||||
|
||||
// SAFETY: `msg` points at a live iovec over `bytes` and the control
|
||||
// buffer we just filled in.
|
||||
let n = unsafe { libc::sendmsg(sock, &raw const msg, 0) };
|
||||
if n < 0 {
|
||||
return Err(std::io::Error::last_os_error());
|
||||
}
|
||||
Ok(usize::try_from(n).unwrap_or_default())
|
||||
}
|
||||
|
||||
/// Send a request to `hive-priv` with an open file descriptor attached,
|
||||
/// and return the response.
|
||||
///
|
||||
/// The helper receives the descriptor itself — not a path or an address
|
||||
/// — so it can act on something it was handed without being told what
|
||||
/// that thing is or how to reach it. Used for
|
||||
/// [`PrivRequest::SendAgentSnapshotToFd`], where the descriptor is a
|
||||
/// socket already connected to a peer hive's snapshot store.
|
||||
///
|
||||
/// ⚠️ Takes the descriptor by value and closes it as soon as the kernel
|
||||
/// has it, *before* awaiting the response. That is not tidiness: a
|
||||
/// socket stays open until every copy of it is closed, so a caller
|
||||
/// holding one back would leave the receiving end waiting for an EOF
|
||||
/// that never comes — `btrfs receive` blocks, and this side reports
|
||||
/// success for a transfer the peer has not committed. Passing ownership
|
||||
/// makes that mistake unrepresentable.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Fails if the socket is unreachable, the descriptor cannot be
|
||||
/// attached, or hive-priv answers with something other than a terminal
|
||||
/// event.
|
||||
pub async fn call_with_fd(req: &PrivRequest, fd: OwnedFd) -> Result<PrivResponse> {
|
||||
let mut stream = UnixStream::connect(PRIV_SOCK)
|
||||
.await
|
||||
.context("connect to hive-priv socket (fd-passing)")?;
|
||||
let line = serde_json::to_string(req).context("serialise PrivRequest")? + "\n";
|
||||
let bytes = line.as_bytes();
|
||||
|
||||
let sock = stream.as_raw_fd();
|
||||
let raw_fd = fd.as_raw_fd();
|
||||
let sent = stream
|
||||
.async_io(Interest::WRITABLE, || send_with_fd(sock, bytes, raw_fd))
|
||||
.await
|
||||
.context("send request + descriptor to hive-priv")?;
|
||||
// The kernel has duplicated the descriptor into hive-priv's queue,
|
||||
// so our copy has done its job. Close it now, before waiting on the
|
||||
// response: see the EOF note on this function.
|
||||
drop(fd);
|
||||
|
||||
// A short sendmsg is legal; the descriptor went with the first
|
||||
// call, so the tail is an ordinary write.
|
||||
if sent < bytes.len() {
|
||||
stream
|
||||
.write_all(&bytes[sent..])
|
||||
.await
|
||||
.context("send remainder of request to hive-priv")?;
|
||||
}
|
||||
stream.shutdown().await.context("shutdown write half")?;
|
||||
|
||||
let mut resp_line = String::new();
|
||||
BufReader::new(stream)
|
||||
.read_line(&mut resp_line)
|
||||
.await
|
||||
.context("read response from hive-priv")?;
|
||||
match serde_json::from_str::<PrivEvent>(&resp_line).context("parse PrivResponse")? {
|
||||
PrivEvent::Done(resp) => Ok(resp),
|
||||
PrivEvent::Line(_) => bail!("unexpected stream line from non-streaming priv op"),
|
||||
}
|
||||
}
|
||||
|
||||
/// Send a streaming request to `hive-priv`, calling `on_line` for each
|
||||
/// `PrivEvent::Line` as it arrives, then returning the terminal
|
||||
/// `PrivResponse`. Used for long-running ops (`create` / `update`).
|
||||
|
|
|
|||
Loading…
Reference in a new issue