mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

remove the old snapshot store; tombstone its options

Browse source
This commit is contained in:
atlas 2026-11-09 23:13:18 +01:00 • committed by mara
commit 20d286f06a
27 changed files with 97 additions and 1308 deletions
Download patch file Download diff file Expand all files Collapse all files

1
README.md
View file

@ -56,7 +56,6 @@ the swarm — one of each, each on whichever host you choose
│
└── opt-in, not part of singleHostSwarm
├── swarm-ci-runner Forgejo Actions runner (deploy.ci-runner)
├── swarm-snapshot-store btrfs-receive endpoint for agent snapshots
└── wg-hive WireGuard mesh between hives (deploy.wireguard)
the hives — the substrate, as many as you like (deploy.hive-controller.enable)

5
docs/2-handbook/agents/persistence.md
View file

@ -657,10 +657,7 @@ qgroup accounting hive-wide (opt-in, no-op on non-btrfs hosts), and
subvolume usage through the same hive-priv-mediated path as
subvolume creation/deletion above.
This is the same subvolume `hivectl agent <name> subvol snapshot push`
sends to the swarm's snapshot store — see
[`docs/4-implementation/networking/snapshot-store.md`](../../4-implementation/networking/snapshot-store.md) for what a pushed
snapshot contains and how the store authenticates a sender.
The old snapshot store no longer exists; swarm-archive will replace it for agent-state backups, and nothing copies the subvolume off the host in the meantime.
The subvolume is the whole agent state root, so `work/` is part of every
snapshot and migration of it and can make them much larger.

26
docs/2-handbook/hives/hivectl-cli.md
View file

@ -25,7 +25,6 @@ This document contains the help content for the `hivectl` command-line program.
* [`hivectl agent subvol snapshot create`↴](#hivectl-agent-subvol-snapshot-create)
* [`hivectl agent subvol snapshot delete`↴](#hivectl-agent-subvol-snapshot-delete)
* [`hivectl agent subvol snapshot send`↴](#hivectl-agent-subvol-snapshot-send)
* [`hivectl agent subvol snapshot push`↴](#hivectl-agent-subvol-snapshot-push)
* [`hivectl list-agents`↴](#hivectl-list-agents)
* [`hivectl quota-enable`↴](#hivectl-quota-enable)
* [`hivectl wg`↴](#hivectl-wg)
@ -273,8 +272,7 @@ Read-only snapshots of this agent's state subvolume
* `create` — Create a read-only snapshot (agent must already be a subvolume)
* `delete` — Delete a snapshot created by `subvol snapshot create`
* `send` — Export a snapshot to a local file via `btrfs send` — the local-file half of the inter-hive migration transport (`push` is the network half). Also useful standalone as a point-in-time backup: a full send with no `--parent` produces a self-contained archive of the snapshot
* `push` — Stream a snapshot to the swarm's snapshot store over the WireGuard mesh — the network half of the migration transport
* `send` — Export a snapshot to a local file via `btrfs send`: a point-in-time backup. A full send with no `--parent` produces a self-contained archive of the snapshot
@ -304,7 +302,7 @@ Delete a snapshot created by `subvol snapshot create`
## `hivectl agent subvol snapshot send`
Export a snapshot to a local file via `btrfs send` — the local-file half of the inter-hive migration transport (`push` is the network half). Also useful standalone as a point-in-time backup: a full send with no `--parent` produces a self-contained archive of the snapshot
Export a snapshot to a local file via `btrfs send`: a point-in-time backup. A full send with no `--parent` produces a self-contained archive of the snapshot
**Usage:** `hivectl agent subvol snapshot send [OPTIONS] --dest <DEST> <LABEL>`
@ -319,26 +317,6 @@ Export a snapshot to a local file via `btrfs send` — the local-file half of th
## `hivectl agent subvol snapshot push`
Stream a snapshot to the swarm's snapshot store over the WireGuard mesh — the network half of the migration transport.
Stages nothing locally: `btrfs send` writes straight into the connection, so a multi-gigabyte agent needs no scratch space on this host. The mesh is the authentication (cryptokey routing binds the sender's address to its key), so there is no credential to pass here.
No destination argument exists: a swarm has one store, read from `services.hyperhive.swarm.snapshotStore`.
**Usage:** `hivectl agent subvol snapshot push [OPTIONS] <LABEL>`
###### **Arguments:**
* `<LABEL>` — Snapshot label passed to `subvol snapshot create --label`
###### **Options:**
* `--parent <PARENT>` — Optional parent snapshot label for an incremental send (`btrfs send -p`) — must be an existing, older snapshot of the same agent, and must already be present on the receiver. Omit for a full send
## `hivectl list-agents`
Show all managed agents with their status and technical state.

6
docs/2-handbook/swarm/README.md
View file

@ -55,9 +55,7 @@ grafana, the controller, authelia, …) →
[`bao.md`](../../4-implementation/swarm/bao.md)
- **swarm CA** — the store root every hive's gateway leaf chains to, and what to
hand a peer. → [`ca.md`](../../4-implementation/swarm/ca.md)
- **snapshot store** — the swarm's one `btrfs receive` endpoint,
`swarm.snapshotStore.{address,port}`. →
[`snapshot-store.md`](../../4-implementation/networking/snapshot-store.md)
- **agent-state backups** — the old snapshot store no longer exists; swarm-archive will replace it.
## Swarm controller
@ -728,8 +726,6 @@ If both hosts are behind NAT, you need a STUN relay or a third host
## Cross-references
- [`ui.md`](ui.md) — the swarm UI, the operator surface for "what hives exist"
- [`4-implementation/networking/snapshot-store.md`](../../4-implementation/networking/snapshot-store.md) — the
swarm's `btrfs receive` endpoint and the `swarm.snapshotStore` option
- [`5-project/process/conventions.md`](../../5-project/process/conventions.md) § Hive identity —
env vars, qualified labels
- [`2-handbook/swarm/matrix.md`](matrix.md) — matrix

240
docs/4-implementation/networking/snapshot-store.md
View file

@ -1,240 +0,0 @@
# Snapshot store
The swarm's `btrfs receive` endpoint. Hives push agent snapshots to it
over the WireGuard mesh; a destination hive later pulls one back to
complete a migration.
Two things it's not, both worth stating because both are common
assumptions:
- **It's not the swarm controller**, and doesn't depend on one. It's
a NixOS host role: a btrfs subvolume tree, a socket-activated
receiver, and the `wg-hive` interface the swarm module already brings
up. That's why you can deploy it before any controller exists.
- **It's not a backup product.** It happens to hold the data a backup
would hold, and you should operate it accordingly (see
[Operating it](#operating-it)) --- but nothing in it does scheduling,
verification, or restore orchestration.
## Enabling it
```nix
services.hyperhive = {
snapshotStore = {
enable = true;
path = "/var/lib/hyperhive-snapshots"; # must be on btrfs
port = 51821;
};
# The mesh is a hard requirement, and is asserted:
deploy.wireguard = {
enable = true;
address = "10.100.0.9/24";
privateKeyFile = "/etc/wireguard/hive.key";
};
};
```
The store host is a swarm member like any other: it gets an entry in
`services.hyperhive.swarm.hives`, the same directory every host holds. See
[swarm/](../../2-handbook/swarm/README.md) for the mesh itself.
Note that `deploy.wireguard.enable` gates the mesh, **not**
`deploy.hive-controller.enable` --- a store host runs no hive and would otherwise get no
`wg-hive` interface at all.
## Pointing a hive at it
The block above configures the host that _receives_. You must tell every
hive that _pushes_ separately where the store is:
```nix
services.hyperhive.swarm.snapshotStore = {
address = "10.100.0.9"; # the store's mesh address, no prefix
port = 51821; # optional; must match the receiver's port
};
```
Two deliberate asymmetries in that pair, both liable to be misread as
inconsistency:
- **`address` has no default.** It's a deployment fact a pushing hive
can't derive, and a wrong guess means streaming an agent's state at
whatever happens to answer. Unset, a push fails naming this option.
- **`port` does default** (`51821`), because it's a convention both
ends read from the same option docs --- a default there is
coordination, not a guess.
Note the option lives under `swarm.*` while the receiving host's lives
under `services.hyperhive.snapshotStore`. That's the distinction the
two namespaces carry throughout: `swarm.*` describes _the swarm_ as seen
from this host, and a bare `services.hyperhive.<service>` describes _a
role this host performs_. A store host sets both --- one to run the
receiver, one only if it also runs a hive that pushes.
With it set, `hivectl agent <name> subvol snapshot push <label>
[--parent <label>]` streams a snapshot straight into the store. No
destination argument exists, because a swarm has exactly one store (see
[One subvolume per agent, not per hive](#one-subvolume-per-agent-not-per-hive)),
and no credential argument, because the mesh is the authentication.
## The mesh is the authentication
No certificates live here, and no key material of its own. That's
deliberate rather than an omission.
WireGuard's cryptokey routing already binds a peer's source address to
its public key: the swarm module configures each peer with
`allowedIPs = [ peer.wireguardAddress ]`, so a packet arriving from
that address provably came from the holder of that private key. By the
time a packet reaches the receiver, the kernel has already
authenticated it.
Layering TLS client certs on top would authenticate _the same fact_ a
second time, and add a credential with an expiry --- a migration that
fails because a renewal quietly didn't happen, discovered on the day
you need to move an agent.
## One subvolume per agent, not per hive
The receiver keys the destination by **agent**.
This isn't cosmetic. After a migration, an agent's next incremental
send arrives from a _different_ hive than the previous one. Keying by
hive would split that agent's snapshot chain across two directories,
and `btrfs send -p` would fail to find its parent --- breaking exactly
the case the store exists to serve.
## What the sender can and can't choose
A `btrfs send` stream carries no notion of _which agent_ it belongs to,
and the subvolume name inside it's chosen by the sender. The
protocol is one `agent <name>` header line, then the raw stream.
The receiver waits 30 s for the header, and systemd stops a transfer
that runs past 6 h in total. The sender gives connect and header write 15 s
each; it puts no limit on the stream itself.
The rule that matters:
> **The receiver owns the destination root. The sender-supplied name is
> validated, never used as a path.**
Validation is an allowlist --- `[A-Za-z0-9_-]+` and nothing else. No
slash and no dot means neither directory traversal nor an absolute path
can survive it. It's deliberately an allowlist and not a list of
forbidden characters: a blocklist only ever excludes the attacks
somebody already thought of.
## Reachability
The receiver is socket-activated, and the socket binds **this host's
mesh address**, never a wildcard. The module asserts three things: the
mesh is on, the address isn't empty, and it isn't `0.0.0.0` or `::` ---
none is just documentation: bound to a wildcard this socket is an
unauthenticated remote write into agent state.
The socket sets `FreeBind`, so the bind succeeds before `wg-hive` has
its address. Nothing orders the wireguard unit before `sockets.target`,
and without it the bind fails with `EADDRNOTAVAIL` at boot and the
receiver never listens.
Binding isn't sufficient on its own. NixOS's firewall is default-deny
and filters in netfilter, _before_ a packet reaches a bound socket, so
the module explicitly opens the port --- scoped to the mesh interface:
```nix
networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ cfg.port ];
```
A host-wide `allowedTCPPorts` would open the port on every interface
including a public NIC, leaving only the socket's bind address between
the internet and a root `btrfs receive`.
## Operating it
### Confinement is the deployment's job
`btrfs receive` needs `CAP_SYS_ADMIN`, so the receiver runs as root.
The unit sets `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp` and a
narrow `ReadWritePaths` --- but those are **defence in depth, not a
boundary**: a process holding `CAP_SYS_ADMIN` can call `mount(2)` and
undo the namespace they set up.
The boundary is the machine. The intended deployments are:
- **a swarm**: the store is its own small VM. The machine is the
boundary, which is stronger than anything the unit could assert about
itself.
- **all-in-one / local**: the store runs as a container on the c0re
host.
The second is worth keeping deliberately, and not only for
convenience: it means every local deployment exercises the confined
path. The usual failure mode for an isolated variant is that
nobody runs it day to day, so it rots and turns up broken in
production.
⚠️ **The assumption to keep true over time:** the store host runs
nothing else. That's true on day one and quietly false the day someone
notices the box has spare disk. Nothing in the config objects when it
stops being true.
### It holds every agent's state from every hive
Which makes it the highest-value target in the swarm by a wide margin,
and means it should get the treatment a backup host gets --- restricted
access, and a decision (rather than an omission) on encryption at rest.
The pitfall is the label: this box holds backup-grade data while nobody
calls it a backup, so it can end up with backup-grade _exposure_ and
non-backup-grade _controls_. Nobody puts a migration staging area on
the access-review list.
### What a snapshot contains
The snapshot covers an agent's **state subvolume**, which is the parent
of `state/`, `claude/` and `harness/` (see
[`docs/2-handbook/agents/persistence.md`'s btrfs subvolume
section](../../2-handbook/agents/persistence.md#btrfs-subvolumes-for-varlibhyperhiveagentsname)
for how and when hive-c0re creates that subvolume). Consequences:
<!-- vale write-good.Passive = NO -->
- The Claude session (`claude/`) travels, so a restored agent keeps its
live `--continue` session rather than needing to log in again.
- `harness/` travels too, including `harness/bash-tasks/`. Task output
is part of an agent's working continuity, so this is wanted --- but it
means anything that has ever leaked into a task's captured output is
in the retained snapshots as well.
<!-- vale write-good.Passive = YES -->
It does **not** cover the agent's config (the forge's `agent-configs/<name>`) or
its topology entry, both of which live outside the subvolume. A restore
therefore yields an agent's memory without its definition.
### Retention
<!-- vale write-good.Passive = NO -->
Retention lives on the _sending_ side (last-N by count, swept
periodically), not here. Count rather than age is deliberate: a count
is bounded by construction, whereas an age policy silently scales disk
usage with how hot a hive runs.
<!-- vale write-good.Passive = YES -->
Per-agent or per-hive `btrfs qgroup` quotas aren't configured yet.
Without them one runaway hive can fill the store and take out every
other hive's snapshots.
## Not built yet
**The pull side.** Push is safe with minimal authorisation because a
hive can only ever write to a chain it owns. Pull is the direction that
needs a policy: unrestricted, any compromised hive could read every
agent's state from every other hive. It needs a notion of which hive
currently owns which agent, and that ownership record lands with the
swarm controller work.
With a single hive the question is trivial --- the only peer owns
everything it sends --- which is why the receive half ships first.

2
docs/4-implementation/trust-boundary/hive-c0re-privilege.md
View file

@ -57,7 +57,7 @@ known operations; there is no arbitrary command pass-through:
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). The caller connects to the destination and hands over the connected socket, so hive-priv never learns an address or protocol; hive-c0re sends no such request today. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
**Strings that reach root-loaded config** — `WriteResourceLimits` and
`WriteNspawnFlags` copy request strings into a systemd drop-in and an env

3
docs/README.md
View file

@ -149,9 +149,6 @@ Nothing of it sits in this tree.
- **What does the matrix homeserver's module assert at evaluation, how
fluffychat-web gets built, and what replaced the hive-side matrix code?** →
[`4-implementation/networking/matrix-homeserver.md`](4-implementation/networking/matrix-homeserver.md).
- **Where do agent snapshots go? How does the swarm's `btrfs receive`
endpoint authenticate a pushing hive?** →
[`4-implementation/networking/snapshot-store.md`](4-implementation/networking/snapshot-store.md).
### Swarm services

1
hive-c0re/src/main.rs
View file

@ -33,7 +33,6 @@ mod paths;
mod priv_client;
mod secret_file;
mod server;
mod snapshot_push;
mod socket_server;
mod stats;
mod stores;

216
hive-c0re/src/priv_client.rs
View file

@ -12,10 +12,9 @@ use hive_priv_sock::{
NetworkIsolation, PRIV_SOCK, PrivEvent, PrivRequest, PrivResponse, PrivStream,
};
use hive_types::NixBuilds;
use std::os::fd::{AsRawFd as _, OwnedFd, RawFd};
use std::path::Path;
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader, Interest};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::UnixStream;
/// Send a single request to `hive-priv` and return the response.
@ -48,144 +47,6 @@ async fn call_at(sock: &Path, req: &PrivRequest) -> Result<PrivResponse> {
}
}
/// Ancillary-data buffer sized and aligned for one `SCM_RIGHTS`
/// message. `CMSG_SPACE` is not a `const fn`, so the size is a literal
/// with room to spare; the union member supplies the `cmsghdr`
/// alignment `CMSG_FIRSTHDR` requires.
#[repr(C)]
union CmsgSpace {
_align: libc::cmsghdr,
bytes: [u8; 32],
}
/// `sendmsg` `bytes` with `fd` attached as `SCM_RIGHTS`, returning how
/// many bytes were accepted.
///
/// The descriptor rides on this one call — ancillary data cannot be
/// sent separately from payload — so the caller must not have written
/// any of `bytes` beforehand.
fn send_with_fd(sock: RawFd, bytes: &[u8], fd: RawFd) -> std::io::Result<usize> {
const FD_SIZE: usize = std::mem::size_of::<RawFd>();
let mut iov = libc::iovec {
iov_base: bytes.as_ptr().cast::<libc::c_void>().cast_mut(),
iov_len: bytes.len(),
};
let mut cmsg = CmsgSpace { bytes: [0; 32] };
// SAFETY: msghdr is a plain C struct with no invalid bit patterns;
// every field we rely on is set immediately below.
let mut msg: libc::msghdr = unsafe { std::mem::zeroed() };
msg.msg_iov = &raw mut iov;
msg.msg_iovlen = 1;
msg.msg_control = std::ptr::addr_of_mut!(cmsg.bytes).cast();
// SAFETY: CMSG_SPACE is a pure size computation.
let space = unsafe { libc::CMSG_SPACE(u32::try_from(FD_SIZE).unwrap_or(4)) };
msg.msg_controllen = space as _;
// SAFETY: the control buffer is live, aligned, and long enough for
// the header CMSG_SPACE just sized.
let hdr = unsafe { libc::CMSG_FIRSTHDR(&raw const msg) };
if hdr.is_null() {
return Err(std::io::Error::other(
"control buffer too small for SCM_RIGHTS",
));
}
// SAFETY: CMSG_LEN is a pure size computation; `hdr` points into
// our own buffer, and write_unaligned tolerates its alignment.
unsafe {
let len = libc::CMSG_LEN(u32::try_from(FD_SIZE).unwrap_or(4));
std::ptr::write_unaligned(
hdr,
libc::cmsghdr {
cmsg_len: len as _,
cmsg_level: libc::SOL_SOCKET,
cmsg_type: libc::SCM_RIGHTS,
},
);
// Copied in byte-wise: the control buffer is only cmsghdr-
// aligned, so casting CMSG_DATA to a *mut RawFd would be
// unsound even where it happens to work.
std::ptr::copy_nonoverlapping(
std::ptr::from_ref(&fd).cast::<u8>(),
libc::CMSG_DATA(hdr),
FD_SIZE,
);
}
// SAFETY: `msg` points at a live iovec over `bytes` and the control
// buffer we just filled in.
let n = unsafe { libc::sendmsg(sock, &raw const msg, 0) };
if n < 0 {
return Err(std::io::Error::last_os_error());
}
Ok(usize::try_from(n).unwrap_or_default())
}
/// Send a request to `hive-priv` with an open file descriptor attached,
/// and return the response.
///
/// The helper receives the descriptor itself — not a path or an address
/// — so it can act on something it was handed without being told what
/// that thing is or how to reach it. Used for
/// [`PrivRequest::SendAgentSnapshotToFd`], where the descriptor is a
/// socket already connected to a peer hive's snapshot store.
///
/// ⚠️ Takes the descriptor by value and closes it as soon as the kernel
/// has it, *before* awaiting the response. That is not tidiness: a
/// socket stays open until every copy of it is closed, so a caller
/// holding one back would leave the receiving end waiting for an EOF
/// that never comes — `btrfs receive` blocks, and this side reports
/// success for a transfer the peer has not committed. Passing ownership
/// makes that mistake unrepresentable.
///
/// # Errors
///
/// Fails if the socket is unreachable, the descriptor cannot be
/// attached, or hive-priv answers with something other than a terminal
/// event.
pub async fn call_with_fd(req: &PrivRequest, fd: OwnedFd) -> Result<PrivResponse> {
call_with_fd_at(Path::new(PRIV_SOCK), req, fd).await
}
async fn call_with_fd_at(path: &Path, req: &PrivRequest, fd: OwnedFd) -> Result<PrivResponse> {
let mut stream = UnixStream::connect(path)
.await
.context("connect to hive-priv socket (fd-passing)")?;
let line = serde_json::to_string(req).context("serialise PrivRequest")? + "\n";
let bytes = line.as_bytes();
let sock = stream.as_raw_fd();
let raw_fd = fd.as_raw_fd();
let sent = stream
.async_io(Interest::WRITABLE, || send_with_fd(sock, bytes, raw_fd))
.await
.context("send request + descriptor to hive-priv")?;
// The kernel has duplicated the descriptor into hive-priv's queue,
// so our copy has done its job. Close it now, before waiting on the
// response: see the EOF note on this function.
drop(fd);
// A short sendmsg is legal; the descriptor went with the first
// call, so the tail is an ordinary write.
if sent < bytes.len() {
stream
.write_all(&bytes[sent..])
.await
.context("send remainder of request to hive-priv")?;
}
stream.shutdown().await.context("shutdown write half")?;
let mut resp_line = String::new();
BufReader::new(stream)
.read_line(&mut resp_line)
.await
.context("read response from hive-priv")?;
match serde_json::from_str::<PrivEvent>(&resp_line).context("parse PrivResponse")? {
PrivEvent::Done(resp) => Ok(resp),
PrivEvent::Line(_) => bail!("unexpected stream line from non-streaming priv op"),
}
}
/// Send a streaming request to `hive-priv`, calling `on_line` for each
/// `PrivEvent::Line` as it arrives, then returning the terminal
/// `PrivResponse`. Used for long-running ops (`create` / `update`).
@ -727,16 +588,14 @@ fn ok(resp: PrivResponse) -> Result<()> {
mod tests {
use super::{
BindMount, CredentialMount, HiveFacts, NetworkIsolation, NixBuilds, PrivRequest,
PrivResponse, PrivStream, call_at, call_streaming_at, call_with_fd_at, check,
create_container_request, older_hive_priv_refused, parse_qgroup_usage,
resource_limits_request, send_snapshot_request, update_container_request, within,
write_nspawn_flags_request,
PrivResponse, PrivStream, call_at, call_streaming_at, check, create_container_request,
older_hive_priv_refused, parse_qgroup_usage, resource_limits_request,
send_snapshot_request, update_container_request, within, write_nspawn_flags_request,
};
use anyhow::Context as _;
use serde_json::{Value, json};
use std::io::{Read as _, Write as _};
use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd, RawFd};
use std::os::unix::net::{UnixListener, UnixStream};
use std::os::unix::net::UnixListener;
use std::time::Duration;
fn wire(req: &PrivRequest) -> Value {
@ -1043,69 +902,4 @@ mod tests {
assert!(err.to_string().contains("before sending Done"), "{err}");
server.join().expect("server");
}
/// `recvmsg` the request and the one descriptor that rides with it.
fn recv_with_fd(conn: &UnixStream) -> (Vec<u8>, OwnedFd) {
let mut buf = [0_u8; 256];
let mut iov = libc::iovec {
iov_base: buf.as_mut_ptr().cast(),
iov_len: buf.len(),
};
// u64 elements give the control buffer the 8-byte alignment a
// `cmsghdr` read needs; a byte array would be misaligned.
let mut cmsg = [0_u64; 8];
// SAFETY: an all-zero msghdr is valid; every pointer set below
// refers to a buffer that outlives the recvmsg call.
let (n, fd) = unsafe {
let mut msg: libc::msghdr = std::mem::zeroed();
msg.msg_iov = &raw mut iov;
msg.msg_iovlen = 1;
msg.msg_control = cmsg.as_mut_ptr().cast();
msg.msg_controllen = std::mem::size_of_val(&cmsg);
let n = libc::recvmsg(conn.as_raw_fd(), &raw mut msg, 0);
assert!(n > 0, "recvmsg failed");
let hdr = libc::CMSG_FIRSTHDR(&raw const msg);
assert!(!hdr.is_null(), "no ancillary data arrived");
assert_eq!((*hdr).cmsg_level, libc::SOL_SOCKET);
assert_eq!((*hdr).cmsg_type, libc::SCM_RIGHTS);
let fd = std::ptr::read_unaligned(libc::CMSG_DATA(hdr).cast::<RawFd>());
(n, OwnedFd::from_raw_fd(fd))
};
(buf[..usize::try_from(n).expect("positive")].to_vec(), fd)
}
#[tokio::test]
async fn a_descriptor_arrives_with_the_request_and_is_closed_on_our_side() {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join("priv.sock");
let listener = UnixListener::bind(&path).expect("bind");
let server = std::thread::spawn(move || {
let (mut conn, _) = listener.accept().expect("accept");
let (request, fd) = recv_with_fd(&conn);
let mut rest = String::new();
conn.read_to_string(&mut rest).expect("read to EOF");
conn.write_all(DONE.as_bytes()).expect("reply");
(request, fd)
});
let (mut read_end, write_end) = std::io::pipe().expect("pipe");
let resp = call_with_fd_at(&path, &PrivRequest::DaemonReload, write_end.into())
.await
.expect("answered");
assert!(resp.ok);
let (request, received) = server.join().expect("server");
assert_eq!(request, b"{\"op\":\"daemon_reload\"}\n");
// The descriptor that arrived is the pipe's write end, and our
// copy is already closed: writing through the received one and
// then dropping it is what ends the read.
let mut write_end = std::fs::File::from(received);
write_end.write_all(b"ping").expect("write via received fd");
drop(write_end);
let mut out = String::new();
read_end
.read_to_string(&mut out)
.expect("EOF after both ends closed");
assert_eq!(out, "ping");
}
}

19
hive-c0re/src/server.rs
View file

@ -202,11 +202,6 @@ async fn dispatch(req: &HostRequest, coord: Arc<Coordinator>) -> HostResponse {
parent,
dest,
} => handle_send_snapshot(name.as_str(), label, parent.as_deref(), dest).await?,
HostRequest::PushSnapshot {
name,
label,
parent,
} => handle_push_snapshot(name.as_str(), label, parent.as_deref()).await?,
})
}
.await;
@ -420,20 +415,6 @@ async fn handle_send_snapshot(
Ok(HostResponse::messages(vec![path]))
}
/// Push a snapshot to the swarm's store. The network sibling of
/// [`handle_send_snapshot`]: nothing lands on this host, so success is
/// bare rather than a path.
async fn handle_push_snapshot(
name: &str,
label: &str,
parent: Option<&str>,
) -> Result<HostResponse> {
crate::snapshot_push::push_agent_snapshot(name, label, parent)
.await
.with_context(|| format!("push {name} snapshot (label {label:?}) to the swarm store"))?;
Ok(HostResponse::success())
}
/// Single-agent queue verbs the admin socket exposes. Each submits the
/// matching DAG (persisting the `wanted` intent, serializing on the
/// agent's lease, with the transient/crash-watch suppression the old

285
hive-c0re/src/snapshot_push.rs
View file

@ -1,285 +0,0 @@
//! Push an agent snapshot to the swarm's snapshot store.
//!
//! This is the c0re half of the transport: it resolves the store,
//! opens the connection, writes the protocol header, and hands the
//! connected socket to `hive-priv`, which runs `btrfs send` straight
//! into it.
//!
//! The split is the point. Everything that requires knowing *where* the
//! store is and *what* the wire protocol looks like happens here, in the
//! unprivileged daemon. hive-priv only ever receives an already-open
//! descriptor, so the root helper never learns an address, never parses
//! a peer list, and never decides who to trust — and once `btrfs send`
//! starts, neither process is in the data path.
//!
//! There is exactly **one** store per swarm, not one per peer hive. The
//! receiver keys destinations by *agent*, so an agent that migrates
//! between hives keeps a single unbroken incremental chain; per-hive
//! stores would split that chain in two, which is the case the store
//! exists to serve.
use std::os::fd::OwnedFd;
use std::time::Duration;
use anyhow::{Context as _, Result, bail};
use hive_priv_sock::PrivRequest;
use tokio::io::AsyncWriteExt as _;
use tokio::net::TcpStream;
use crate::priv_client;
/// `host:port` of the swarm's snapshot store, rendered by
/// `nix/host-modules/hive-c0re/environment.nix` from
/// `services.hyperhive.swarm.snapshotStore`.
const STORE_ENV: &str = "HYPERHIVE_SNAPSHOT_STORE";
/// Deadline for the TCP connect to the store. An unreachable mesh address
/// otherwise waits out the kernel's own connect timeout while holding the
/// admin-socket handler.
const CONNECT_TIMEOUT: Duration = Duration::from_secs(15);
/// Deadline for writing and flushing the one header line into a connected
/// socket. Shorter than the receiver's own header deadline.
const HEADER_WRITE_TIMEOUT: Duration = Duration::from_secs(15);
/// Send `snapshot` of `agent` to the swarm's snapshot store, optionally
/// as an incremental against `parent`.
///
/// # Errors
///
/// Fails when no store is configured, it is unreachable, or hive-priv
/// reports the `btrfs send` failed.
pub async fn push_agent_snapshot(agent: &str, snapshot: &str, parent: Option<&str>) -> Result<()> {
let configured = std::env::var(STORE_ENV).ok();
let addr = require_store_addr(configured.as_deref())?;
let sock = open_stream(addr, agent, CONNECT_TIMEOUT, HEADER_WRITE_TIMEOUT).await?;
// Hand the connected socket over. `call_with_fd` takes ownership and
// closes our copy as soon as the kernel has it, so the receiver sees
// EOF when `btrfs send` finishes rather than hanging on a descriptor
// this process is still holding.
let fd = detach_blocking(sock)?;
let resp = priv_client::call_with_fd(
&PrivRequest::SendAgentSnapshotToFd {
agent_name: agent.to_owned(),
snapshot_name: snapshot.to_owned(),
parent_snapshot_name: parent.map(str::to_owned),
},
fd,
)
.await
.with_context(|| format!("stream {agent}/{snapshot} to the swarm snapshot store"))?;
if !resp.ok {
bail!(
"push of {agent}/{snapshot} failed: {}",
resp.error.unwrap_or_else(|| resp.stderr.clone())
);
}
tracing::info!(
agent,
snapshot,
parent,
store = addr,
"pushed agent snapshot to the swarm snapshot store"
);
Ok(())
}
/// Connect to the store and write the protocol header.
///
/// The header names which agent this stream belongs to: a btrfs stream
/// carries the sender's own subvolume name, not the hive's notion of the
/// agent, so the receiver cannot infer it. One line, because the receiver
/// reads it with `read` and must not buffer past the newline into the byte
/// stream. Only the connect and the header are bounded; the stream body that
/// follows is `btrfs send` writing straight into the socket and may take as
/// long as the snapshot is large.
async fn open_stream(
addr: &str,
agent: &str,
connect_limit: Duration,
header_limit: Duration,
) -> Result<TcpStream> {
let mut sock = bounded(
connect_limit,
"connect to the swarm snapshot store",
TcpStream::connect(addr),
)
.await?
.with_context(|| format!("connect to the swarm snapshot store at {addr}"))?;
bounded(
header_limit,
"send header to the swarm snapshot store",
async {
sock.write_all(format!("agent {agent}\n").as_bytes())
.await?;
sock.flush().await
},
)
.await?
.context("send header to the swarm snapshot store")?;
Ok(sock)
}
/// Run `fut` for at most `limit`; a timeout is an error naming `what`.
async fn bounded<T>(limit: Duration, what: &str, fut: impl Future<Output = T>) -> Result<T> {
tokio::time::timeout(limit, fut)
.await
.map_err(|_| anyhow::anyhow!("{what} timed out after {limit:?}"))
}
/// Turn a tokio socket into a descriptor safe to hand to `btrfs send`.
///
/// tokio sockets are `O_NONBLOCK`, and `into_std` does not clear it. The
/// flag lives on the open file description, which `hive-priv` shares with
/// us through `SCM_RIGHTS`, so `btrfs send` would write to a
/// non-blocking socket and get `EAGAIN` whenever the send buffer is full.
fn detach_blocking(sock: TcpStream) -> Result<OwnedFd> {
let std_sock = sock
.into_std()
.context("detach snapshot-store socket for hand-off")?;
std_sock
.set_nonblocking(false)
.context("switch snapshot-store socket to blocking mode for hand-off")?;
Ok(std_sock.into())
}
/// Require a configured store address.
///
/// Split out of [`push_agent_snapshot`] so the "not configured" path is
/// testable without setting a process-wide env var, which would race
/// other tests.
///
/// # Errors
///
/// Fails when unset, naming the option to set. There is deliberately no
/// fallback: an address is a deployment fact this host cannot derive,
/// and guessing one means pushing an agent's state at whatever happens
/// to answer.
fn require_store_addr(configured: Option<&str>) -> Result<&str> {
match configured {
Some(addr) if !addr.is_empty() => Ok(addr),
_ => bail!(
"no snapshot store configured for this swarm — set \
services.hyperhive.swarm.snapshotStore.address (and .port if it \
differs from the default) on this host"
),
}
}
#[cfg(test)]
mod tests {
use std::os::fd::AsRawFd as _;
use tokio::net::{TcpListener, TcpStream};
use std::time::Duration;
use super::{
CONNECT_TIMEOUT, HEADER_WRITE_TIMEOUT, bounded, detach_blocking, open_stream,
require_store_addr,
};
#[tokio::test]
async fn the_handed_off_descriptor_is_blocking() {
let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind");
let sock = TcpStream::connect(listener.local_addr().expect("addr"))
.await
.expect("connect");
let fd = detach_blocking(sock).expect("detach");
// SAFETY: `fd` is a live descriptor owned by this test.
let flags = unsafe { libc::fcntl(fd.as_raw_fd(), libc::F_GETFL) };
assert!(flags >= 0, "F_GETFL failed");
assert_eq!(
flags & libc::O_NONBLOCK,
0,
"btrfs send would get EAGAIN on a full socket buffer"
);
}
#[tokio::test]
async fn a_future_that_never_finishes_errors_at_the_deadline() {
let err = bounded(
Duration::from_millis(50),
"connect",
std::future::pending::<()>(),
)
.await
.expect_err("pending must time out");
assert!(
format!("{err:#}").contains("connect timed out after"),
"{err:#}"
);
}
#[tokio::test]
async fn a_header_the_peer_never_drains_errors_at_the_deadline() {
// The peer accepts and holds the connection without reading, so the
// header cannot drain once the socket buffers fill; a name this
// large exceeds any loopback buffer.
let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind");
let addr = listener.local_addr().expect("addr").to_string();
let peer = tokio::spawn(async move {
let (held, _) = listener.accept().await.expect("accept");
std::future::pending::<()>().await;
drop(held);
});
let huge = "a".repeat(128 * 1024 * 1024);
let err = open_stream(&addr, &huge, CONNECT_TIMEOUT, Duration::from_millis(200))
.await
.expect_err("an undrained header must time out");
peer.abort();
let msg = format!("{err:#}");
assert!(
msg.contains("send header to the swarm snapshot store timed out after"),
"{msg}"
);
}
#[tokio::test]
async fn a_reachable_peer_gets_the_header() {
use tokio::io::AsyncReadExt as _;
let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind");
let addr = listener.local_addr().expect("addr").to_string();
let sock = open_stream(&addr, "alice", CONNECT_TIMEOUT, HEADER_WRITE_TIMEOUT)
.await
.expect("open");
let (mut peer, _) = listener.accept().await.expect("accept");
drop(sock);
let mut got = String::new();
peer.read_to_string(&mut got).await.expect("read");
assert_eq!(got, "agent alice\n");
}
#[test]
fn a_configured_address_passes_through() {
assert_eq!(
require_store_addr(Some("10.100.0.1:51821")).expect("configured"),
"10.100.0.1:51821"
);
}
#[test]
fn an_unconfigured_store_names_the_option_and_never_guesses() {
let err = require_store_addr(None).expect_err("no store means no push");
let msg = format!("{err:#}");
assert!(msg.contains("swarm.snapshotStore.address"), "{msg}");
// A guessed address or port is the failure this is guarding
// against: it would push an agent's state at whatever answers.
assert!(!msg.contains("51821"), "must not suggest a value: {msg}");
}
#[test]
fn an_empty_value_is_treated_as_unset() {
// An empty env var is a misconfiguration, not an address; it
// must fail the same way rather than reaching connect().
let err = require_store_addr(Some("")).expect_err("empty is not an address");
assert!(format!("{err:#}").contains("no snapshot store configured"));
}
}

17
hive-host-sock/src/lib.rs
View file

@ -263,23 +263,6 @@ pub enum HostRequest {
parent: Option<String>,
dest: String,
},
/// Push a snapshot to the swarm's snapshot store over the WireGuard
/// mesh (`hivectl agent <name> subvol snapshot push`). The network
/// sibling of [`HostRequest::SendSnapshot`]: same snapshot and
/// optional incremental `parent`, but the stream goes to the store's
/// receiver instead of a local file.
///
/// There is no destination field: a swarm has exactly one store, and
/// the daemon reads its address from
/// `services.hyperhive.swarm.snapshotStore`, failing if none is
/// configured. Bare success — nothing is written on this host to
/// report a path for.
PushSnapshot {
name: Ident,
label: String,
#[serde(default)]
parent: Option<String>,
},
}
/// One agent's btrfs qgroup usage row — the [`HostRequest::QuotaShow`]

7
hive-priv-sock/src/lib.rs
View file

@ -650,13 +650,12 @@ pub enum PrivRequest {
/// ancillary data on the same socket): `btrfs send [-p <parent>]
/// <snapshot> >&<passed fd>`.
///
/// The network half of the inter-hive migration transport. hive-c0re
/// connects to the peer hive's snapshot store, writes the header
/// itself, and hands the **connected socket** over — so hive-priv
/// The caller connects to the destination, writes any header itself,
/// and hands the **connected socket** over — so hive-priv
/// never learns an address, a protocol, or that a network is
/// involved, and nobody sits in the data path once the send starts
/// (which is what makes a multi-gigabyte transfer survive a
/// hive-c0re restart).
/// hive-c0re restart). hive-c0re sends no such request today.
///
/// Exactly one descriptor must accompany this request. hive-priv
/// rejects the request if none arrived, if more than one did, or if a

7
hive-priv/src/storage.rs
View file

@ -957,10 +957,9 @@ async fn send_agent_snapshot_to_file(
/// incremental against `parent_name`) straight into a descriptor the
/// caller passed us.
///
/// The network half of the inter-hive migration transport, arranged so
/// this helper never learns there *is* a network: hive-c0re connects to
/// the peer's snapshot store, writes the header itself, and hands the
/// connected socket over. We only ever see "a thing to write bytes into",
/// Arranged so this helper never learns there *is* a network: the
/// caller connects to the destination, writes any header itself, and
/// hands the connected socket over. We only ever see "a thing to write bytes into",
/// which keeps a root process out of any address, protocol or trust
/// decision — and keeps everyone out of the data path once `btrfs send`
/// starts, which matters at multi-gigabyte sizes.

27
hivectl/src/cli.rs
View file

@ -341,10 +341,8 @@ pub enum SnapshotCmd {
/// Snapshot label passed to `subvol snapshot create --label`.
label: String,
},
/// Export a snapshot to a local file via `btrfs send` — the
/// local-file half of the inter-hive migration transport (`push`
/// is the network half). Also useful standalone as a point-in-time
/// backup: a full send with no `--parent` produces a
/// Export a snapshot to a local file via `btrfs send`: a point-in-time
/// backup. A full send with no `--parent` produces a
/// self-contained archive of the snapshot.
Send {
/// Snapshot label passed to `subvol snapshot create --label`.
@ -359,25 +357,4 @@ pub enum SnapshotCmd {
#[arg(long)]
dest: String,
},
/// Stream a snapshot to the swarm's snapshot store over the
/// WireGuard mesh — the network half of the migration transport.
///
/// Stages nothing locally: `btrfs send` writes straight into the
/// connection, so a multi-gigabyte agent needs no scratch space on
/// this host. The mesh is the authentication (cryptokey routing
/// binds the sender's address to its key), so there is no credential
/// to pass here.
///
/// No destination argument exists: a swarm has one store, read
/// from `services.hyperhive.swarm.snapshotStore`.
Push {
/// Snapshot label passed to `subvol snapshot create --label`.
label: String,
/// Optional parent snapshot label for an incremental send
/// (`btrfs send -p`) — must be an existing, older snapshot of the
/// same agent, and must already be present on the receiver.
/// Omit for a full send.
#[arg(long)]
parent: Option<String>,
},
}

37
hivectl/src/subvol.rs
View file

@ -41,9 +41,6 @@ pub(crate) async fn dispatch_subvol(socket: &Path, name: &str, cmd: SubvolCmd) -
parent,
dest,
} => subvol_snapshot_send(socket, name, &label, parent.as_deref(), &dest).await,
SnapshotCmd::Push { label, parent } => {
subvol_snapshot_push(socket, name, &label, parent.as_deref()).await
}
},
}
}
@ -217,37 +214,3 @@ async fn subvol_snapshot_send(
)
.await
}
/// `subvol snapshot push <agent> <label> [--parent <label>]` — stream a
/// snapshot to the swarm's snapshot store over the mesh.
///
/// The network sibling of `send`. Nothing is staged on this host, so
/// there is no path to print: success is silent apart from the
/// confirmation below. No destination argument — a swarm has one store,
/// and the daemon reads its address from the host config.
async fn subvol_snapshot_push(
socket: &Path,
name: &str,
label: &str,
parent: Option<&str>,
) -> Result<()> {
daemon_request(
socket,
hive_host_sock::HostRequest::PushSnapshot {
name: crate::util::parse_ident(name)?,
label: label.to_owned(),
parent: parent.map(str::to_owned),
},
"snapshot push",
)
.await?;
match parent {
Some(p) => {
println!(
"pushed {name} snapshot {label:?} (incremental from {p:?}) to the swarm store"
);
}
None => println!("pushed {name} snapshot {label:?} (full) to the swarm store"),
}
Ok(())
}

4
nix/checks.nix
View file

@ -203,10 +203,6 @@ in
inherit pkgs self nixosSystem;
inherit (pkgs) lib;
};
module-eval-snapshot-store = import ./module-eval/snapshot-store.nix {
inherit pkgs self nixosSystem;
inherit (pkgs) lib;
};
module-eval-agent-queue-bao = import ./module-eval/agent-queue-bao.nix {
inherit pkgs self nixosSystem;
inherit (pkgs) lib;

1
nix/host-modules/default.nix
View file

@ -85,7 +85,6 @@
./swarm-grafana.nix
./swarm-otel-service.nix
./swarm-otel.nix
./swarm-snapshot-store.nix
./swarm-ui.nix
./swarm-victorialogs-service.nix
./swarm-victorialogs.nix

38
nix/host-modules/deploy.nix
View file

@ -68,6 +68,12 @@ let
there is no agent-base closure to pre-fetch into the host store. Remove
this definition.
'';
# Shared by the removed options of the snapshot store.
snapshotStoreRemoved = ''
The snapshot store was removed; agent-state backups are being rebuilt as
swarm-archive. Remove this definition.
'';
in
{
imports = [
@ -143,6 +149,38 @@ in
"queue"
"agentCredentialDir"
] queueAgentCredentialRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"swarm"
"snapshotStore"
"address"
] snapshotStoreRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"swarm"
"snapshotStore"
"port"
] snapshotStoreRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"snapshotStore"
"enable"
] snapshotStoreRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"snapshotStore"
"path"
] snapshotStoreRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"snapshotStore"
"port"
] snapshotStoreRemoved)
# Same type, same meaning, new path — so a rename carries it exactly
# and existing configs keep evaluating with one warning naming both

12
nix/host-modules/hive-c0re/environment.nix
View file

@ -199,18 +199,6 @@ in
# set (no browser-reachable matrix vhost).
HIVE_MATRIX_PUBLIC_URL = "https://${config.services.hyperhive.swarm.matrix.gatewayHost}/";
}
// lib.optionalAttrs (config.services.hyperhive.swarm.snapshotStore.address != null) {
# `host:port` of the swarm's single snapshot store, for pushing agent
# snapshots (hive-c0re::snapshot_push). One per swarm, not one per
# peer — the receiver keys destinations by agent so a migrating agent
# keeps one incremental chain. Absent when no store is configured, and
# a push then fails naming the option rather than guessing.
HYPERHIVE_SNAPSHOT_STORE =
let
s = config.services.hyperhive.swarm.snapshotStore;
in
"${s.address}:${toString s.port}";
}
//
# Swarm-queue coordinates for offering this hive's status upward
# (hive-c0re::swarm_status). All three together or none: a half-set

Some files were not shown because too many files have changed in this diff Show more