Compare commits

...
Author SHA1 Message Date
atlas
7b23b53b75 docs(#2862): document the push side and regenerate the CLI reference
docs/tools/hivectl-cli.md is generated by `hivectl markdown-docs` and
diffed against a fresh run by the hivectl-docs flake check, so adding
the push verb without regenerating it would have failed CI. The
regeneration also retires two copies of a sentence describing the
cross-hive leg as an ssh pipe that "isn't wired up yet" -- ssh was
dropped when the WireGuard mesh became the authentication, and the leg
is wired up now.

snapshot-store.md documented only the receiving host, so a reader had
no way to learn how a pushing hive is told where the store is. It now
covers services.hyperhive.swarm.snapshotStore, including why address
has no default and port does: an address is a deployment fact that
cannot be guessed, a port is a convention both ends read from the same
docs. It also states the namespace rule the two options illustrate --
swarm.* describes the swarm as seen from here, a bare
services.hyperhive.<service> describes a role this host performs.

swarm.md never mentioned the store even though the option lives in its
namespace, so a reader configuring swarm.peers had no signal it exists.
2026-07-31 22:15:37 +02:00
atlas
ba71e45486 refactor(#2862): one snapshot store per swarm, not one per peer
The push side modelled a store per peer hive: a --peer argument, a
swarm.peers.<domain>.snapshotStorePort option, and a swarm_peers module
whose entire job was answering "which peer". A swarm has exactly one
store, so none of that had anything to select between.

The receiver already proved it. It keys destination directories by
agent, not by sending hive, precisely so an agent that migrates keeps
one unbroken incremental chain -- which only makes sense if every hive
pushes to the same place. Per-hive stores would split the chain in two,
the case that keying exists to prevent.

So the destination moves to services.hyperhive.swarm.snapshotStore,
rendered into HYPERHIVE_SNAPSHOT_STORE, and swarm_peers is deleted
rather than adapted. address has no default because it is a
deployment fact this host cannot derive; port defaults because it is a
convention both ends read from the same option docs. An unset or empty
address fails naming the option instead of connecting somewhere
arbitrary, and a test asserts the message suggests no value.
2026-07-31 22:15:37 +02:00
atlas
258c0998ab test(#2862): cover both fd/op mismatch branches in hive-priv
check_fd_agreement is the guard that keeps a descriptor and the request
it arrived with in agreement, and both of its rejections were untested.

An fd-taking op with no descriptor must not fall back to anything: a
temp file or the response socket would send an agent's state somewhere
the caller never asked for. The mirror case matters for a different
reason -- returning the error is what drops the OwnedFd and closes it,
so ignoring a stray descriptor instead would leak one per bad request
in a long-lived root process.

The third test pins both agreeing combinations, so the check is
rejecting mismatches rather than descriptors in general. Descriptors
are real /dev/null handles so the closing drop is genuinely exercised.
2026-07-31 22:15:37 +02:00
atlas
282bbc3709 feat(#2862): push a snapshot to a peer hive's store over the mesh
Adds the caller the fd-passing machinery existed for: hivectl agent
<name> subvol snapshot push --peer <hive> resolves the peer, connects
to its snapshot store, writes the agent header, and hands the connected
socket to hive-priv, which runs btrfs send straight into it.

The split keeps the root helper ignorant. Everything that involves
knowing where a peer is, what the wire protocol looks like, and which
hive to trust happens in the unprivileged daemon; hive-priv only ever
receives an already-open descriptor. Once btrfs send starts, neither
process is in the data path, so a multi-gigabyte transfer costs no
per-byte work and survives a hive-c0re restart.

call_with_fd takes the descriptor by value and closes it as soon as the
kernel has it. A socket stays open until every copy closes, so holding
one back would leave the receiver waiting for an EOF that never comes:
btrfs receive blocks and this side reports success for a transfer the
peer never committed. Ownership makes that unrepresentable.

The peer's store port is a new swarm.peers.<domain>.snapshotStorePort
option rather than a constant matching the module default. A pushing
hive cannot read the receiver's configuration, so assuming 51821 would
push at a port nobody promised to listen on; absent, the push fails
naming the option. swarm_peers parses the mesh address the host module
has always rendered into HYPERHIVE_PEERS but nothing read.
2026-07-31 22:15:37 +02:00
atlas
51f352f0ca feat(#2862): receive a passed descriptor and stream a snapshot into it
hive-priv read requests with BufReader::lines, which cannot surface
SCM_RIGHTS: ancillary data is attached to one specific recvmsg call, so
a buffered line reader takes the bytes and silently drops the
descriptor. Replace it with a recvmsg loop.

The pairing is deliberately trivial. hive-sock-client connects per
request, so a connection carries one line and at most one descriptor;
a second descriptor arriving before its line is a protocol error rather
than something to queue. check_fd_agreement rejects both mismatches --
an fd-taking op that got none, and a descriptor sent to an op that
takes none -- and dropping the OwnedFd on that path closes it.

recv_with_fds claims every descriptor the kernel attaches, including
ones this protocol never expects, because an fd we fail to claim leaks
for the life of the process. MSG_CMSG_CLOEXEC keeps a received
descriptor out of every btrfs and nixos-container child. The control
buffer is only cmsghdr-aligned, so descriptors are copied out
byte-wise instead of read through a more strictly aligned pointer.

SendAgentSnapshotToFd is SendAgentSnapshotToFile without the staging
file: same validation and -p parent handling, stdout wired to the
passed descriptor. It exists so hive-c0re can connect to a peer hive's
snapshot store, write the header itself, and hand over the connected
socket -- leaving this helper with no address, no protocol, and nobody
in the data path once the send starts.
2026-07-31 22:15:37 +02:00
atlas
364bc290df refactor(#2862): drop the fd framing module, the hazard is unreachable
The Framer bound a passed descriptor to the request line it belongs to,
on the premise that several requests can be in flight on one connection
so a descriptor could arrive with a chunk belonging to a different one.

That premise is false. hive-sock-client::try_once connects per request
(connect, write one line, read one line, drop) and priv_client's two
connect sites each open their own stream, so a connection carries
exactly one request: one line, at most one descriptor, nothing to
disambiguate. Request and response align by connection.

Delete it rather than move it. The recvmsg swap still has to happen —
SCM_RIGHTS is attached to a specific recvmsg call and BufReader::lines
cannot surface it — but the pairing it needs is "take the descriptor
that arrived with this line", not a queue and a claim policy.
2026-07-31 22:15:37 +02:00
atlas
ec4ba4c7fa feat(#2862): fd-carrying line framing for the priv socket
First half of the fd-passing work, and deliberately the half with the
real failure mode in it. No syscalls here — the caller does the
recvmsg and feeds this (bytes, fds); it hands back complete messages
paired with the descriptor each one owns.

Association is the whole point. A descriptor does not arrive neatly
paired with the request that wants it: recvmsg returns whatever bytes
happen to be available plus whatever ancillary data rode along, so a
descriptor can arrive with a chunk holding only part of its request's
line, with a chunk whose bytes finish the previous request, ahead of
any of its own bytes, or alongside several complete requests at once.

Pairing "the fd from this chunk" with "the request in this chunk" is
therefore wrong in the worst way: the types are identical either way,
so nothing catches it, and the failure is one request executing
against another's descriptor — in this process, writing one agent's
state into a different transfer's socket. So descriptors queue on
arrival and each message claims the oldest unclaimed one at the moment
it completes.

Two consequences worth stating: a line that fails to decode does NOT
consume a descriptor (closing it there would destroy something
belonging to a request nobody processed), and unclaimed descriptors
are drainable so the teardown path can close them instead of leaking
one per abandoned message in a long-lived helper.

Lives in hive-priv-sock, not hive-priv: clippy's dead-code error was
right that an unwired module doesn't belong in the binary, and chasing
that produced the better home anyway — both ends need this. The daemon
sends descriptors and the helper reassembles them, so framing is part
of the wire contract rather than one side's implementation detail.
2026-07-31 22:15:37 +02:00
14 changed files with 884 additions and 30 deletions

View file

@ -41,6 +41,41 @@ Note that the mesh is gated on `swarm.wireguard.enable`, **not** on
`c0re.enable` --- a store host runs no hive and would otherwise get no
`wg-hive` interface at all.
## Pointing a hive at it
The block above configures the host that *receives*. Every hive that
*pushes* separately needs to be told where the store is:
```nix
services.hyperhive.swarm.snapshotStore = {
address = "10.100.0.9"; # the store's mesh address, no prefix
port = 51821; # optional; must match the receiver's port
};
```
Two deliberate asymmetries in that pair, both easy to misread as
inconsistency:
- **`address` has no default.** It is a deployment fact a pushing hive
cannot derive, and a wrong guess means streaming an agent's state at
whatever happens to answer. Unset, a push fails naming this option.
- **`port` does default** (`51821`), because it is a convention both
ends read from the same option docs --- a default there is
coordination, not a guess.
Note the option lives under `swarm.*` while the receiving host's lives
under `services.hyperhive.snapshotStore`. That is the distinction the
two namespaces carry throughout: `swarm.*` describes *the swarm* as seen
from this host, and a bare `services.hyperhive.<service>` describes *a
role this host performs*. A store host sets both --- one to run the
receiver, one only if it also runs a hive that pushes.
With it set, `hivectl agent <name> subvol snapshot push <label>
[--parent <label>]` streams a snapshot straight into the store. There is
no destination argument, because a swarm has exactly one store (see
[One subvolume per agent, not per hive](#one-subvolume-per-agent-not-per-hive)),
and no credential argument, because the mesh is the authentication.
## The mesh is the authentication
There are no certificates here, and no key material of its own. That is

View file

@ -221,8 +221,22 @@ other side initiates. With keepalive on, the NAT hole stays open.
If both hosts are behind NAT, a STUN relay or a third host (exit node)
is required. Out of scope for v0.
## Snapshot store
One further option lives in this namespace but is documented with the
service it points at: `services.hyperhive.swarm.snapshotStore.{address,
port}` tells this hive where the swarm's `btrfs receive` endpoint is, so
`hivectl agent <name> subvol snapshot push` has somewhere to stream to.
It is genuinely swarm-scoped rather than per-peer — a swarm has exactly
one store, because the receiver keys destinations by *agent* so a
migrating agent keeps one unbroken incremental chain. See
[snapshot-store.md](snapshot-store.md).
## Cross-references
- `docs/snapshot-store.md` — the swarm's `btrfs receive` endpoint, and
the `swarm.snapshotStore` option that points a hive at it
- `docs/conventions.md` § Hive identity — env vars, qualified labels
- `docs/matrix.md` — matrix federation, TLS cert auto-generation,
firewall posture

View file

@ -42,6 +42,7 @@ This document contains the help content for the `hivectl` command-line program.
* [`hivectl agent subvol snapshot create`↴](#hivectl-agent-subvol-snapshot-create)
* [`hivectl agent subvol snapshot delete`↴](#hivectl-agent-subvol-snapshot-delete)
* [`hivectl agent subvol snapshot send`↴](#hivectl-agent-subvol-snapshot-send)
* [`hivectl agent subvol snapshot push`↴](#hivectl-agent-subvol-snapshot-push)
* [`hivectl list-agents`↴](#hivectl-list-agents)
* [`hivectl quota-enable`↴](#hivectl-quota-enable)
* [`hivectl approvals`↴](#hivectl-approvals)
@ -562,7 +563,8 @@ Read-only snapshots of this agent's state subvolume
* `create` — Create a read-only snapshot (agent must already be a subvolume)
* `delete` — Delete a snapshot created by `subvol snapshot create`
* `send` — Export a snapshot to a local file via `btrfs send` (the local-file half of inter-hive migration transport; the cross-hive `ssh ... btrfs receive` leg isn't wired up yet). Also useful standalone as a point-in-time backup: a full send with no `--parent` produces a self-contained archive of the snapshot
* `send` — Export a snapshot to a local file via `btrfs send` — the local-file half of the inter-hive migration transport (`push` is the network half). Also useful standalone as a point-in-time backup: a full send with no `--parent` produces a self-contained archive of the snapshot
* `push` — Stream a snapshot to the swarm's snapshot store over the WireGuard mesh — the network half of the migration transport
@ -592,7 +594,7 @@ Delete a snapshot created by `subvol snapshot create`
## `hivectl agent subvol snapshot send`
Export a snapshot to a local file via `btrfs send` (the local-file half of inter-hive migration transport; the cross-hive `ssh ... btrfs receive` leg isn't wired up yet). Also useful standalone as a point-in-time backup: a full send with no `--parent` produces a self-contained archive of the snapshot
Export a snapshot to a local file via `btrfs send` — the local-file half of the inter-hive migration transport (`push` is the network half). Also useful standalone as a point-in-time backup: a full send with no `--parent` produces a self-contained archive of the snapshot
**Usage:** `hivectl agent subvol snapshot send [OPTIONS] --dest <DEST> <LABEL>`
@ -607,6 +609,26 @@ Export a snapshot to a local file via `btrfs send` (the local-file half of inter
## `hivectl agent subvol snapshot push`
Stream a snapshot to the swarm's snapshot store over the WireGuard mesh — the network half of the migration transport.
Nothing is staged locally: `btrfs send` writes straight into the connection, so a multi-gigabyte agent needs no scratch space on this host. The mesh is the authentication (cryptokey routing binds the sender's address to its key), so there is no credential to pass here.
There is no destination argument: a swarm has one store, read from `services.hyperhive.swarm.snapshotStore`.
**Usage:** `hivectl agent subvol snapshot push [OPTIONS] <LABEL>`
###### **Arguments:**
* `<LABEL>` — Snapshot label passed to `subvol snapshot create --label`
###### **Options:**
* `--parent <PARENT>` — Optional parent snapshot label for an incremental send (`btrfs send -p`) — must be an existing, older snapshot of the same agent, and must already be present on the receiver. Omit for a full send
## `hivectl list-agents`
Show all managed agents with their status and technical state.

View file

@ -31,6 +31,7 @@ mod paths;
mod priv_client;
mod questions;
mod server;
mod snapshot_push;
mod socket_server;
mod stats;
mod stores;

View file

@ -11,7 +11,9 @@ use hive_priv_sock::{
BindMount, CredentialMount, InfraAction, InfraContainer, JournalQuery, NetworkIsolation,
PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream,
};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use std::os::fd::{AsRawFd as _, OwnedFd, RawFd};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader, Interest};
use tokio::net::UnixStream;
/// Send a single request to `hive-priv` and return the response.
@ -40,6 +42,140 @@ pub async fn call(req: &PrivRequest) -> Result<PrivResponse> {
}
}
/// Ancillary-data buffer sized and aligned for one `SCM_RIGHTS`
/// message. `CMSG_SPACE` is not a `const fn`, so the size is a literal
/// with room to spare; the union member supplies the `cmsghdr`
/// alignment `CMSG_FIRSTHDR` requires.
#[repr(C)]
union CmsgSpace {
_align: libc::cmsghdr,
bytes: [u8; 32],
}
/// `sendmsg` `bytes` with `fd` attached as `SCM_RIGHTS`, returning how
/// many bytes were accepted.
///
/// The descriptor rides on this one call — ancillary data cannot be
/// sent separately from payload — so the caller must not have written
/// any of `bytes` beforehand.
fn send_with_fd(sock: RawFd, bytes: &[u8], fd: RawFd) -> std::io::Result<usize> {
const FD_SIZE: usize = std::mem::size_of::<RawFd>();
let mut iov = libc::iovec {
iov_base: bytes.as_ptr().cast::<libc::c_void>().cast_mut(),
iov_len: bytes.len(),
};
let mut cmsg = CmsgSpace { bytes: [0; 32] };
// SAFETY: msghdr is a plain C struct with no invalid bit patterns;
// every field we rely on is set immediately below.
let mut msg: libc::msghdr = unsafe { std::mem::zeroed() };
msg.msg_iov = &raw mut iov;
msg.msg_iovlen = 1;
msg.msg_control = std::ptr::addr_of_mut!(cmsg.bytes).cast();
// SAFETY: CMSG_SPACE is a pure size computation.
let space = unsafe { libc::CMSG_SPACE(u32::try_from(FD_SIZE).unwrap_or(4)) };
msg.msg_controllen = space as _;
// SAFETY: the control buffer is live, aligned, and long enough for
// the header CMSG_SPACE just sized.
let hdr = unsafe { libc::CMSG_FIRSTHDR(&raw const msg) };
if hdr.is_null() {
return Err(std::io::Error::other(
"control buffer too small for SCM_RIGHTS",
));
}
// SAFETY: CMSG_LEN is a pure size computation; `hdr` points into
// our own buffer, and write_unaligned tolerates its alignment.
unsafe {
let len = libc::CMSG_LEN(u32::try_from(FD_SIZE).unwrap_or(4));
std::ptr::write_unaligned(
hdr,
libc::cmsghdr {
cmsg_len: len as _,
cmsg_level: libc::SOL_SOCKET,
cmsg_type: libc::SCM_RIGHTS,
},
);
// Copied in byte-wise: the control buffer is only cmsghdr-
// aligned, so casting CMSG_DATA to a *mut RawFd would be
// unsound even where it happens to work.
std::ptr::copy_nonoverlapping(
std::ptr::from_ref(&fd).cast::<u8>(),
libc::CMSG_DATA(hdr),
FD_SIZE,
);
}
// SAFETY: `msg` points at a live iovec over `bytes` and the control
// buffer we just filled in.
let n = unsafe { libc::sendmsg(sock, &raw const msg, 0) };
if n < 0 {
return Err(std::io::Error::last_os_error());
}
Ok(usize::try_from(n).unwrap_or_default())
}
/// Send a request to `hive-priv` with an open file descriptor attached,
/// and return the response.
///
/// The helper receives the descriptor itself — not a path or an address
/// — so it can act on something it was handed without being told what
/// that thing is or how to reach it. Used for
/// [`PrivRequest::SendAgentSnapshotToFd`], where the descriptor is a
/// socket already connected to a peer hive's snapshot store.
///
/// ⚠️ Takes the descriptor by value and closes it as soon as the kernel
/// has it, *before* awaiting the response. That is not tidiness: a
/// socket stays open until every copy of it is closed, so a caller
/// holding one back would leave the receiving end waiting for an EOF
/// that never comes — `btrfs receive` blocks, and this side reports
/// success for a transfer the peer has not committed. Passing ownership
/// makes that mistake unrepresentable.
///
/// # Errors
///
/// Fails if the socket is unreachable, the descriptor cannot be
/// attached, or hive-priv answers with something other than a terminal
/// event.
pub async fn call_with_fd(req: &PrivRequest, fd: OwnedFd) -> Result<PrivResponse> {
let mut stream = UnixStream::connect(PRIV_SOCK)
.await
.context("connect to hive-priv socket (fd-passing)")?;
let line = serde_json::to_string(req).context("serialise PrivRequest")? + "\n";
let bytes = line.as_bytes();
let sock = stream.as_raw_fd();
let raw_fd = fd.as_raw_fd();
let sent = stream
.async_io(Interest::WRITABLE, || send_with_fd(sock, bytes, raw_fd))
.await
.context("send request + descriptor to hive-priv")?;
// The kernel has duplicated the descriptor into hive-priv's queue,
// so our copy has done its job. Close it now, before waiting on the
// response: see the EOF note on this function.
drop(fd);
// A short sendmsg is legal; the descriptor went with the first
// call, so the tail is an ordinary write.
if sent < bytes.len() {
stream
.write_all(&bytes[sent..])
.await
.context("send remainder of request to hive-priv")?;
}
stream.shutdown().await.context("shutdown write half")?;
let mut resp_line = String::new();
BufReader::new(stream)
.read_line(&mut resp_line)
.await
.context("read response from hive-priv")?;
match serde_json::from_str::<PrivEvent>(&resp_line).context("parse PrivResponse")? {
PrivEvent::Done(resp) => Ok(resp),
PrivEvent::Line(_) => bail!("unexpected stream line from non-streaming priv op"),
}
}
/// Send a streaming request to `hive-priv`, calling `on_line` for each
/// `PrivEvent::Line` as it arrives, then returning the terminal
/// `PrivResponse`. Used for long-running ops (`create` / `update`).

View file

@ -264,6 +264,11 @@ async fn dispatch(req: &HostRequest, coord: Arc<Coordinator>) -> HostResponse {
parent,
dest,
} => handle_send_snapshot(name.as_str(), label, parent.as_deref(), dest).await?,
HostRequest::PushSnapshot {
name,
label,
parent,
} => handle_push_snapshot(name.as_str(), label, parent.as_deref()).await?,
})
}
.await;
@ -661,6 +666,20 @@ async fn handle_send_snapshot(
Ok(HostResponse::messages(vec![path]))
}
/// Push a snapshot to the swarm's store. The network sibling of
/// [`handle_send_snapshot`]: nothing lands on this host, so success is
/// bare rather than a path.
async fn handle_push_snapshot(
name: &str,
label: &str,
parent: Option<&str>,
) -> Result<HostResponse> {
crate::snapshot_push::push_agent_snapshot(name, label, parent)
.await
.with_context(|| format!("push {name} snapshot (label {label:?}) to the swarm store"))?;
Ok(HostResponse::success())
}
async fn handle_matrix_sync_admin() -> Result<HostResponse> {
require_matrix_present().await?;
let register_token =

View file

@ -0,0 +1,148 @@
//! Push an agent snapshot to the swarm's snapshot store.
//!
//! This is the c0re half of the transport: it resolves the store,
//! opens the connection, writes the protocol header, and hands the
//! connected socket to `hive-priv`, which runs `btrfs send` straight
//! into it.
//!
//! The split is the point. Everything that requires knowing *where* the
//! store is and *what* the wire protocol looks like happens here, in the
//! unprivileged daemon. hive-priv only ever receives an already-open
//! descriptor, so the root helper never learns an address, never parses
//! a peer list, and never decides who to trust — and once `btrfs send`
//! starts, neither process is in the data path.
//!
//! There is exactly **one** store per swarm, not one per peer hive. The
//! receiver keys destinations by *agent*, so an agent that migrates
//! between hives keeps a single unbroken incremental chain; per-hive
//! stores would split that chain in two, which is the case the store
//! exists to serve.
use anyhow::{Context as _, Result, bail};
use hive_priv_sock::PrivRequest;
use tokio::io::AsyncWriteExt as _;
use tokio::net::TcpStream;
use crate::priv_client;
/// `host:port` of the swarm's snapshot store, rendered by
/// `nix/host-modules/hive-c0re/environment.nix` from
/// `services.hyperhive.swarm.snapshotStore`.
const STORE_ENV: &str = "HYPERHIVE_SNAPSHOT_STORE";
/// Send `snapshot` of `agent` to the swarm's snapshot store, optionally
/// as an incremental against `parent`.
///
/// # Errors
///
/// Fails when no store is configured, it is unreachable, or hive-priv
/// reports the `btrfs send` failed.
pub async fn push_agent_snapshot(agent: &str, snapshot: &str, parent: Option<&str>) -> Result<()> {
let configured = std::env::var(STORE_ENV).ok();
let addr = require_store_addr(configured.as_deref())?;
let mut sock = TcpStream::connect(addr)
.await
.with_context(|| format!("connect to the swarm snapshot store at {addr}"))?;
// The header names which agent this stream belongs to: a btrfs
// stream carries the sender's own subvolume name, not the hive's
// notion of the agent, so the receiver cannot infer it. One line,
// because the receiver reads it with `read` and must not buffer
// past the newline into the byte stream.
sock.write_all(format!("agent {agent}\n").as_bytes())
.await
.context("send header to the swarm snapshot store")?;
sock.flush()
.await
.context("flush header to the swarm snapshot store")?;
// Hand the connected socket over. `call_with_fd` takes ownership and
// closes our copy as soon as the kernel has it, so the receiver sees
// EOF when `btrfs send` finishes rather than hanging on a descriptor
// this process is still holding.
let fd = sock
.into_std()
.context("detach snapshot-store socket for hand-off")?
.into();
let resp = priv_client::call_with_fd(
&PrivRequest::SendAgentSnapshotToFd {
agent_name: agent.to_owned(),
snapshot_name: snapshot.to_owned(),
parent_snapshot_name: parent.map(str::to_owned),
},
fd,
)
.await
.with_context(|| format!("stream {agent}/{snapshot} to the swarm snapshot store"))?;
if !resp.ok {
bail!(
"push of {agent}/{snapshot} failed: {}",
resp.error.unwrap_or_else(|| resp.stderr.clone())
);
}
tracing::info!(
agent,
snapshot,
parent,
store = addr,
"pushed agent snapshot to the swarm snapshot store"
);
Ok(())
}
/// Require a configured store address.
///
/// Split out of [`push_agent_snapshot`] so the "not configured" path is
/// testable without setting a process-wide env var, which would race
/// other tests.
///
/// # Errors
///
/// Fails when unset, naming the option to set. There is deliberately no
/// fallback: an address is a deployment fact this host cannot derive,
/// and guessing one means pushing an agent's state at whatever happens
/// to answer.
fn require_store_addr(configured: Option<&str>) -> Result<&str> {
match configured {
Some(addr) if !addr.is_empty() => Ok(addr),
_ => bail!(
"no snapshot store configured for this swarm — set \
services.hyperhive.swarm.snapshotStore.address (and .port if it \
differs from the default) on this host"
),
}
}
#[cfg(test)]
mod tests {
use super::require_store_addr;
#[test]
fn a_configured_address_passes_through() {
assert_eq!(
require_store_addr(Some("10.100.0.1:51821")).expect("configured"),
"10.100.0.1:51821"
);
}
#[test]
fn an_unconfigured_store_names_the_option_and_never_guesses() {
let err = require_store_addr(None).expect_err("no store means no push");
let msg = format!("{err:#}");
assert!(msg.contains("swarm.snapshotStore.address"), "{msg}");
// A guessed address or port is the failure this is guarding
// against: it would push an agent's state at whatever answers.
assert!(!msg.contains("51821"), "must not suggest a value: {msg}");
}
#[test]
fn an_empty_value_is_treated_as_unset() {
// An empty env var is a misconfiguration, not an address; it
// must fail the same way rather than reaching connect().
let err = require_store_addr(Some("")).expect_err("empty is not an address");
assert!(format!("{err:#}").contains("no snapshot store configured"));
}
}

View file

@ -375,6 +375,23 @@ pub enum HostRequest {
parent: Option<String>,
dest: String,
},
/// Push a snapshot to the swarm's snapshot store over the WireGuard
/// mesh (`hivectl agent <name> subvol snapshot push`). The network
/// sibling of [`HostRequest::SendSnapshot`]: same snapshot and
/// optional incremental `parent`, but the stream goes to the store's
/// receiver instead of a local file.
///
/// There is no destination field: a swarm has exactly one store, and
/// the daemon reads its address from
/// `services.hyperhive.swarm.snapshotStore`, failing if none is
/// configured. Bare success — nothing is written on this host to
/// report a path for.
PushSnapshot {
name: Ident,
label: String,
#[serde(default)]
parent: Option<String>,
},
}
/// One agent's btrfs qgroup usage row — the [`HostRequest::QuotaShow`]

View file

@ -715,6 +715,35 @@ pub enum PrivRequest {
dest_file_name: String,
},
/// Stream a previously-created read-only snapshot into a file
/// descriptor the caller passes alongside this request (`SCM_RIGHTS`
/// ancillary data on the same socket): `btrfs send [-p <parent>]
/// <snapshot> >&<passed fd>`.
///
/// The network half of the inter-hive migration transport. hive-c0re
/// connects to the peer hive's snapshot store, writes the header
/// itself, and hands the **connected socket** over — so hive-priv
/// never learns an address, a protocol, or that a network is
/// involved, and nobody sits in the data path once the send starts
/// (which is what makes a multi-gigabyte transfer survive a
/// hive-c0re restart).
///
/// Exactly one descriptor must accompany this request. hive-priv
/// rejects the request if none arrived, if more than one did, or if a
/// descriptor arrives alongside any *other* operation — no guessing
/// when the caller didn't say. Requires root.
SendAgentSnapshotToFd {
/// Logical agent name (validated by `validate_agent_name`).
agent_name: String,
/// Snapshot label to send, same validation as `SnapshotAgentSubvolume`.
snapshot_name: String,
/// Optional parent snapshot label for an incremental
/// (`btrfs send -p`) send — must be an older read-only snapshot of
/// the same agent, still present on disk. `None` sends the full
/// snapshot.
parent_snapshot_name: Option<String>,
},
/// Write `/etc/tmpfiles.d/hyperhive-agents.conf` for the given agent set
/// and immediately apply it with `systemd-tmpfiles --create`. Each entry
/// declares the per-agent runtime dirs (`/run/hyperhive/agents/<name>` and

View file

@ -17,6 +17,7 @@
//! `LISTEN_FDS=1` + `LISTEN_PID=<self>`, the inherited fd 3 is used
//! instead of binding a fresh socket.
use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd, RawFd};
use std::path::{Path, PathBuf};
use anyhow::{Context as _, Result, bail};
@ -26,7 +27,7 @@ use hive_priv_sock::{
PAUSED_MARKER_FILE, PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream,
PrivStreamLine, SIBLING_CONTAINERS,
};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::io::{AsyncWriteExt, BufReader};
use tokio::net::unix::OwnedWriteHalf;
use tokio::net::{UnixListener, UnixStream};
use tokio::process::Command;
@ -92,11 +93,162 @@ fn socket_listener() -> Result<UnixListener> {
Ok(listener)
}
/// Ancillary-data buffer sized and aligned for one `SCM_RIGHTS` message.
///
/// `CMSG_SPACE` is not a `const fn`, so the size is a literal with room
/// to spare (24 bytes are needed for a single descriptor on x86-64). The
/// union member gives the `cmsghdr` alignment `CMSG_FIRSTHDR` requires —
/// a bare `[u8; N]` is only byte-aligned and would be undefined behaviour
/// to walk.
#[repr(C)]
union CmsgSpace {
_align: libc::cmsghdr,
bytes: [u8; 32],
}
/// One `recvmsg` into `buf`, returning the bytes read plus any file
/// descriptors that rode along as `SCM_RIGHTS`.
///
/// Why not a plain read: ancillary data is attached to a *specific*
/// `recvmsg` call, so a buffered line reader cannot surface it — it
/// reads the bytes and silently drops the descriptor.
///
/// `MSG_CMSG_CLOEXEC` is not optional: without it a received descriptor
/// is inherited by every `btrfs` / `nixos-container` child this helper
/// later spawns.
fn recv_with_fds(sock: RawFd, buf: &mut [u8]) -> std::io::Result<(usize, Vec<OwnedFd>)> {
const FD_SIZE: usize = std::mem::size_of::<RawFd>();
let mut iov = libc::iovec {
iov_base: buf.as_mut_ptr().cast(),
iov_len: buf.len(),
};
let mut cmsg = CmsgSpace { bytes: [0; 32] };
// SAFETY: msghdr is a plain C struct with no invalid bit patterns;
// every field we care about is set immediately below.
let mut msg: libc::msghdr = unsafe { std::mem::zeroed() };
msg.msg_iov = &raw mut iov;
msg.msg_iovlen = 1;
msg.msg_control = std::ptr::addr_of_mut!(cmsg.bytes).cast();
msg.msg_controllen = 32;
// SAFETY: `msg` points at a live iovec covering `buf` and a live,
// correctly aligned control buffer of the length we just declared.
let n = unsafe { libc::recvmsg(sock, &raw mut msg, libc::MSG_CMSG_CLOEXEC) };
if n < 0 {
return Err(std::io::Error::last_os_error());
}
// Take ownership of every descriptor the kernel attached, even ones
// this protocol never expects: an `OwnedFd` we drop is closed, an
// fd we fail to claim is leaked for the lifetime of the process.
let mut fds = Vec::new();
// SAFETY: `msg` was just filled in by a successful `recvmsg`.
let mut cmsgp = unsafe { libc::CMSG_FIRSTHDR(&raw const msg) };
while !cmsgp.is_null() {
// SAFETY: CMSG_FIRSTHDR / CMSG_NXTHDR only ever return a pointer
// to a complete header inside the control buffer.
let hdr = unsafe { std::ptr::read_unaligned(cmsgp) };
if hdr.cmsg_level == libc::SOL_SOCKET && hdr.cmsg_type == libc::SCM_RIGHTS {
// SAFETY: same, and CMSG_LEN(0) is the header's own length.
let payload = hdr.cmsg_len as usize - unsafe { libc::CMSG_LEN(0) } as usize;
let count = payload / FD_SIZE;
// SAFETY: CMSG_DATA points at `payload` bytes of descriptors.
let data = unsafe { libc::CMSG_DATA(cmsgp) };
for i in 0..count {
// Copied out byte-wise rather than read through a
// `*const RawFd`: the control buffer is only guaranteed
// `cmsghdr`-aligned, so casting to a more strictly
// aligned pointer would be unsound even where it happens
// to work.
let mut raw = [0u8; FD_SIZE];
// SAFETY: i < count, so this reads inside the payload.
unsafe {
std::ptr::copy_nonoverlapping(data.add(i * FD_SIZE), raw.as_mut_ptr(), FD_SIZE);
}
// SAFETY: the kernel just created this descriptor for
// us — we are its only owner.
fds.push(unsafe { OwnedFd::from_raw_fd(RawFd::from_ne_bytes(raw)) });
}
}
// SAFETY: `cmsgp` came from this same message.
cmsgp = unsafe { libc::CMSG_NXTHDR(&raw const msg, cmsgp) };
}
// `n >= 0` was checked above, so the conversion cannot fail; going
// through `try_from` keeps it a cast-free, lint-clean widening.
let read = usize::try_from(n).unwrap_or_default();
Ok((read, fds))
}
/// Reads newline-delimited requests off one connection, pairing each
/// with the descriptor that arrived with it.
///
/// The pairing is deliberately trivial, because the protocol is:
/// `hive-sock-client` connects per request, so a connection carries one
/// line and at most one descriptor. The loop below still handles several
/// sequential requests (the server always has), but it refuses to guess
/// — a second descriptor arriving before its line is a protocol error,
/// not something to queue and hope about.
struct Requests<'a> {
sock: &'a UnixStream,
buf: Vec<u8>,
fd: Option<OwnedFd>,
}
impl Requests<'_> {
/// Next complete request line and its descriptor, or `None` at EOF.
async fn next(&mut self) -> Result<Option<(String, Option<OwnedFd>)>> {
loop {
if let Some(nl) = self.buf.iter().position(|&b| b == b'\n') {
let line: Vec<u8> = self.buf.drain(..=nl).take(nl).collect();
let line = String::from_utf8(line).context("request line was not valid UTF-8")?;
return Ok(Some((line, self.fd.take())));
}
let mut chunk = [0u8; 8192];
let raw = self.sock.as_raw_fd();
let (n, fds) = self
.sock
.async_io(tokio::io::Interest::READABLE, || {
recv_with_fds(raw, &mut chunk)
})
.await
.context("recvmsg on the priv socket")?;
for fd in fds {
if self.fd.replace(fd).is_some() {
bail!("more than one file descriptor passed for a single request");
}
}
if n == 0 {
if !self.buf.is_empty() {
bail!("connection closed mid-request ({} bytes)", self.buf.len());
}
return Ok(None);
}
self.buf.extend_from_slice(&chunk[..n]);
}
}
}
async fn handle(stream: UnixStream) {
let (reader, mut writer) = stream.into_split();
let mut lines = BufReader::new(reader).lines();
while let Ok(Some(line)) = lines.next_line().await {
let resp = dispatch(&line, &mut writer).await;
let mut requests = Requests {
sock: reader.as_ref(),
buf: Vec::new(),
fd: None,
};
loop {
let (line, fd) = match requests.next().await {
Ok(Some(req)) => req,
Ok(None) => break,
Err(e) => {
tracing::warn!(error = %format!("{e:#}"), "reading request failed");
break;
}
};
let resp = dispatch(&line, fd, &mut writer).await;
// Write the terminal PrivResponse as a PrivEvent::Done. Wire-identical
// to a bare PrivResponse (untagged), so old hive-c0re callers that
// deserialise directly to PrivResponse continue to work.
@ -112,31 +264,54 @@ async fn handle(stream: UnixStream) {
}
}
async fn dispatch(line: &str, writer: &mut OwnedWriteHalf) -> PrivResponse {
match serde_json::from_str::<PrivRequest>(line) {
Ok(req) => match exec(req, writer).await {
Ok((stdout, stderr)) => PrivResponse {
ok: true,
stdout,
stderr,
error: None,
},
Err(e) => PrivResponse {
ok: false,
stdout: String::new(),
stderr: String::new(),
error: Some(format!("{e:#}")),
},
/// Reject a request whose descriptor and operation disagree, in either
/// direction.
///
/// No guessing when the caller didn't say: an op that streams into a
/// passed descriptor cannot invent one, and an op that takes none must
/// not silently accept one. Returning the `Err` here drops the
/// `OwnedFd`, which closes it.
fn check_fd_agreement(req: &PrivRequest, fd: Option<&OwnedFd>) -> Result<()> {
let wants_fd = matches!(req, PrivRequest::SendAgentSnapshotToFd { .. });
match (wants_fd, fd.is_some()) {
(true, false) => bail!("this operation requires a passed file descriptor, none arrived"),
(false, true) => bail!("this operation does not take a passed file descriptor"),
_ => Ok(()),
}
}
async fn dispatch(line: &str, fd: Option<OwnedFd>, writer: &mut OwnedWriteHalf) -> PrivResponse {
match run(line, fd, writer).await {
Ok((stdout, stderr)) => PrivResponse {
ok: true,
stdout,
stderr,
error: None,
},
Err(e) => PrivResponse {
ok: false,
stdout: String::new(),
stderr: String::new(),
error: Some(format!("parse request: {e}")),
error: Some(format!("{e:#}")),
},
}
}
/// Parse one request line, check it agrees with the descriptor that
/// arrived with it, and execute it.
///
/// Split out of [`dispatch`] so the three failure modes collapse into one
/// `Result` instead of three nested matches building the same struct.
async fn run(
line: &str,
fd: Option<OwnedFd>,
writer: &mut OwnedWriteHalf,
) -> Result<(String, String)> {
let req = serde_json::from_str::<PrivRequest>(line).context("parse request")?;
check_fd_agreement(&req, fd.as_ref())?;
exec(req, fd, writer).await
}
/// Write one `PrivEvent::Line` to the client. Best-effort: a write
/// failure is logged but doesn't abort the running subprocess.
async fn write_line_event(writer: &mut OwnedWriteHalf, stream: PrivStream, data: &str) {
@ -159,7 +334,17 @@ async fn write_line_event(writer: &mut OwnedWriteHalf, stream: PrivStream, data:
// One match arm per priv op — a flat 1:1 dispatch table. The length tracks
// the op count, not complexity; splitting it would just scatter the mapping.
#[allow(clippy::too_many_lines)]
async fn exec(req: PrivRequest, writer: &mut OwnedWriteHalf) -> Result<(String, String)> {
/// Execute one validated request.
///
/// `fd` is the descriptor that arrived with this request, already checked
/// against the operation by [`check_fd_agreement`]: `Some` exactly for
/// the variants that stream into a caller-supplied descriptor, `None`
/// for every other operation.
async fn exec(
req: PrivRequest,
fd: Option<OwnedFd>,
writer: &mut OwnedWriteHalf,
) -> Result<(String, String)> {
match req {
PrivRequest::StartContainer { ref name } => {
validate_container_name(name)?;
@ -417,6 +602,26 @@ async fn exec(req: PrivRequest, writer: &mut OwnedWriteHalf) -> Result<(String,
.await
}
PrivRequest::SendAgentSnapshotToFd {
ref agent_name,
ref snapshot_name,
ref parent_snapshot_name,
} => {
validate_agent_name(agent_name)?;
validate_snapshot_name(snapshot_name)?;
if let Some(parent) = parent_snapshot_name {
validate_snapshot_name(parent)?;
}
let dest = fd.context("no descriptor to stream into")?;
send_agent_snapshot_to_fd(
agent_name,
snapshot_name,
parent_snapshot_name.as_deref(),
dest,
)
.await
}
PrivRequest::SyncAgentTmpfiles { ref agents } => sync_agent_tmpfiles(agents).await,
}
}
@ -1424,6 +1629,70 @@ async fn send_agent_snapshot_to_file(
Ok((dest.display().to_string(), String::new()))
}
/// `SendAgentSnapshotToFd` — stream a read-only snapshot (optionally
/// incremental against `parent_name`) straight into a descriptor the
/// caller passed us.
///
/// The network half of the inter-hive migration transport, arranged so
/// this helper never learns there *is* a network: hive-c0re connects to
/// the peer's snapshot store, writes the header itself, and hands the
/// connected socket over. We only ever see "a thing to write bytes into",
/// which keeps a root process out of any address, protocol or trust
/// decision — and keeps everyone out of the data path once `btrfs send`
/// starts, which matters at multi-gigabyte sizes.
async fn send_agent_snapshot_to_fd(
agent_name: &str,
snapshot_name: &str,
parent_name: Option<&str>,
dest: OwnedFd,
) -> Result<(String, String)> {
let snap = snapshot_path(agent_name, snapshot_name);
if !snap.exists() {
bail!(
"snapshot {} does not exist — create it with `subvol snapshot create` first",
snap.display()
);
}
let mut cmd = Command::new("btrfs");
cmd.arg("send");
if let Some(parent) = parent_name {
let parent_path = snapshot_path(agent_name, parent);
if !parent_path.exists() {
bail!(
"parent snapshot {} does not exist — pick an existing parent or omit it for a full send",
parent_path.display()
);
}
cmd.arg("-p").arg(&parent_path);
}
cmd.arg(&snap);
cmd.stdout(std::process::Stdio::from(dest));
cmd.stderr(std::process::Stdio::piped());
let out = cmd
.spawn()
.with_context(|| format!("spawn btrfs send {}", snap.display()))?
.wait_with_output()
.await
.with_context(|| format!("wait on btrfs send {}", snap.display()))?;
if !out.status.success() {
// Nothing to clean up: the destination isn't ours. A partial
// stream is the receiving end's problem, and `btrfs receive`
// refuses to commit an incomplete subvolume anyway.
bail!(
"btrfs send {} failed: {}",
snap.display(),
String::from_utf8_lossy(&out.stderr).trim()
);
}
tracing::info!(
agent = %agent_name, snapshot = %snap.display(), parent = ?parent_name,
"streamed agent snapshot into a passed descriptor"
);
Ok((String::new(), String::new()))
}
/// `SetSubvolumeQuota` — set or clear a qgroup size limit on an agent
/// subvolume (`btrfs qgroup limit <bytes|none> <…/agent_name>`). See the
/// wire doc.
@ -2217,12 +2486,68 @@ async fn sync_agent_tmpfiles(agents: &[String]) -> Result<(String, String)> {
#[cfg(test)]
mod tests {
use super::{
PAUSED_MARKER_FILE, limits_dropin_body, redact_password_line, remove_marker_in,
write_state_file_nofollow,
OwnedFd, PAUSED_MARKER_FILE, PrivRequest, check_fd_agreement, limits_dropin_body,
redact_password_line, remove_marker_in, write_state_file_nofollow,
};
use std::path::PathBuf;
use std::sync::atomic::{AtomicU32, Ordering};
/// A request that streams into a caller-supplied descriptor.
fn fd_taking_request() -> PrivRequest {
PrivRequest::SendAgentSnapshotToFd {
agent_name: "atlas".to_owned(),
snapshot_name: "hive-migrate".to_owned(),
parent_snapshot_name: None,
}
}
/// A real descriptor — `/dev/null` rather than a fake, so the drop
/// that closes it on a rejection path is genuinely exercised.
fn some_fd() -> OwnedFd {
std::fs::File::open("/dev/null")
.expect("open /dev/null")
.into()
}
/// An op that streams into a passed descriptor cannot invent one:
/// falling back to anything (a temp file, the response socket) would
/// send an agent's state somewhere the caller never asked for.
#[test]
fn an_fd_taking_op_without_a_descriptor_is_rejected() {
let err = check_fd_agreement(&fd_taking_request(), None)
.expect_err("no descriptor arrived, so this must not proceed");
let msg = format!("{err:#}");
assert!(msg.contains("requires a passed file descriptor"), "{msg}");
}
/// The mirror case: a descriptor sent alongside an op that takes
/// none is a protocol error, not something to ignore. Returning the
/// error drops the `OwnedFd`, which closes it — the alternative
/// leaks one descriptor per stray request in a long-lived root
/// process.
#[test]
fn a_descriptor_sent_to_an_op_that_takes_none_is_rejected() {
let fd = some_fd();
let err = check_fd_agreement(&PrivRequest::DaemonReload, Some(&fd))
.expect_err("an unexpected descriptor must not be silently ignored");
let msg = format!("{err:#}");
assert!(
msg.contains("does not take a passed file descriptor"),
"{msg}"
);
}
/// Both agreeing combinations pass, so the check rejects mismatches
/// rather than descriptors in general.
#[test]
fn agreeing_combinations_are_accepted() {
let fd = some_fd();
check_fd_agreement(&fd_taking_request(), Some(&fd))
.expect("an fd-taking op with its descriptor is the normal case");
check_fd_agreement(&PrivRequest::DaemonReload, None)
.expect("every ordinary request arrives without a descriptor");
}
/// An unset weight is "not configured": the drop-in must come out
/// byte-identical to the pre-weights two-setting body, so a hive-c0re
/// older than this field can't change what lands on disk.

View file

@ -620,10 +620,10 @@ pub enum SnapshotCmd {
/// Snapshot label passed to `subvol snapshot create --label`.
label: String,
},
/// Export a snapshot to a local file via `btrfs send` (the local-file
/// half of inter-hive migration transport; the cross-hive `ssh ...
/// btrfs receive` leg isn't wired up yet). Also useful standalone as a
/// point-in-time backup: a full send with no `--parent` produces a
/// Export a snapshot to a local file via `btrfs send` — the
/// local-file half of the inter-hive migration transport (`push`
/// is the network half). Also useful standalone as a point-in-time
/// backup: a full send with no `--parent` produces a
/// self-contained archive of the snapshot.
Send {
/// Snapshot label passed to `subvol snapshot create --label`.
@ -638,4 +638,25 @@ pub enum SnapshotCmd {
#[arg(long)]
dest: String,
},
/// Stream a snapshot to the swarm's snapshot store over the
/// WireGuard mesh — the network half of the migration transport.
///
/// Nothing is staged locally: `btrfs send` writes straight into the
/// connection, so a multi-gigabyte agent needs no scratch space on
/// this host. The mesh is the authentication (cryptokey routing
/// binds the sender's address to its key), so there is no credential
/// to pass here.
///
/// There is no destination argument: a swarm has one store, read
/// from `services.hyperhive.swarm.snapshotStore`.
Push {
/// Snapshot label passed to `subvol snapshot create --label`.
label: String,
/// Optional parent snapshot label for an incremental send
/// (`btrfs send -p`) — must be an existing, older snapshot of the
/// same agent, and must already be present on the receiver.
/// Omit for a full send.
#[arg(long)]
parent: Option<String>,
},
}

View file

@ -44,6 +44,9 @@ pub(crate) async fn dispatch_subvol(socket: &Path, name: &str, cmd: SubvolCmd) -
parent,
dest,
} => subvol_snapshot_send(socket, name, &label, parent.as_deref(), &dest).await,
SnapshotCmd::Push { label, parent } => {
subvol_snapshot_push(socket, name, &label, parent.as_deref()).await
}
},
}
}
@ -217,3 +220,37 @@ async fn subvol_snapshot_send(
)
.await
}
/// `subvol snapshot push <agent> <label> [--parent <label>]` — stream a
/// snapshot to the swarm's snapshot store over the mesh.
///
/// The network sibling of `send`. Nothing is staged on this host, so
/// there is no path to print: success is silent apart from the
/// confirmation below. No destination argument — a swarm has one store,
/// and the daemon reads its address from the host config.
async fn subvol_snapshot_push(
socket: &Path,
name: &str,
label: &str,
parent: Option<&str>,
) -> Result<()> {
daemon_request(
socket,
hive_host_sock::HostRequest::PushSnapshot {
name: crate::util::parse_ident(name)?,
label: label.to_owned(),
parent: parent.map(str::to_owned),
},
"snapshot push",
)
.await?;
match parent {
Some(p) => {
println!(
"pushed {name} snapshot {label:?} (incremental from {p:?}) to the swarm store"
);
}
None => println!("pushed {name} snapshot {label:?} (full) to the swarm store"),
}
Ok(())
}

View file

@ -149,6 +149,18 @@ in
# or no gatewayHost is set (no browser-reachable matrix vhost).
HIVE_MATRIX_PUBLIC_URL = "https://${config.services.hyperhive.matrix.gatewayHost}/";
}
// lib.optionalAttrs (config.services.hyperhive.swarm.snapshotStore.address != null) {
# `host:port` of the swarm's single snapshot store, for pushing agent
# snapshots (hive-c0re::snapshot_push). One per swarm, not one per
# peer — the receiver keys destinations by agent so a migrating agent
# keeps one incremental chain. Absent when no store is configured, and
# a push then fails naming the option rather than guessing.
HYPERHIVE_SNAPSHOT_STORE =
let
s = config.services.hyperhive.swarm.snapshotStore;
in
"${s.address}:${toString s.port}";
}
// lib.optionalAttrs (config.services.hyperhive.swarm.peers != { }) {
# Peer hives serialised as a JSON array of {domain, cert_fingerprint,
# wireguard_address?} objects. Consumed by hive-agent::identity::peers()

View file

@ -105,6 +105,7 @@
are silently excluded from `wg-hive`).
'';
};
};
}
);
@ -125,4 +126,41 @@
'';
};
options.services.hyperhive.swarm.snapshotStore = {
address = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "10.100.0.1";
description = ''
Mesh address of the swarm's snapshot store the single
`btrfs receive` endpoint every hive in this swarm pushes agent
snapshots to. Bare IP, no prefix.
There is exactly **one** store per swarm, not one per peer: the
receiver keys destinations by *agent*, so an agent that migrates
between hives keeps a single unbroken incremental chain. Per-hive
stores would split that chain in two, which is the case the store
exists to serve.
Null means this swarm has no store configured, and pushing fails
saying so rather than guessing an address. Set it on every hive
that pushes; the receiving host separately sets
`services.hyperhive.snapshotStore.enable`.
'';
};
port = lib.mkOption {
type = lib.types.port;
default = 51821;
description = ''
TCP port the swarm's snapshot store listens on. Must match the
receiving host's `services.hyperhive.snapshotStore.port`.
Defaulted (unlike `address`) because it is a shared convention
both sides read from the same option docs whereas an address
is deployment-specific and cannot be guessed.
'';
};
};
}