mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

remove the old snapshot store; tombstone its options

Browse source
This commit is contained in:
atlas 2026-11-09 23:13:18 +01:00 • committed by mara
commit 20d286f06a
2 changed files with 97 additions and 1308 deletions
Download patch file Download diff file Expand all files Collapse all files

240
docs/4-implementation/networking/snapshot-store.md
View file

@ -1,240 +0,0 @@
# Snapshot store
The swarm's `btrfs receive` endpoint. Hives push agent snapshots to it
over the WireGuard mesh; a destination hive later pulls one back to
complete a migration.
Two things it's not, both worth stating because both are common
assumptions:
- **It's not the swarm controller**, and doesn't depend on one. It's
a NixOS host role: a btrfs subvolume tree, a socket-activated
receiver, and the `wg-hive` interface the swarm module already brings
up. That's why you can deploy it before any controller exists.
- **It's not a backup product.** It happens to hold the data a backup
would hold, and you should operate it accordingly (see
[Operating it](#operating-it)) --- but nothing in it does scheduling,
verification, or restore orchestration.
## Enabling it
```nix
services.hyperhive = {
snapshotStore = {
enable = true;
path = "/var/lib/hyperhive-snapshots"; # must be on btrfs
port = 51821;
};
# The mesh is a hard requirement, and is asserted:
deploy.wireguard = {
enable = true;
address = "10.100.0.9/24";
privateKeyFile = "/etc/wireguard/hive.key";
};
};
```
The store host is a swarm member like any other: it gets an entry in
`services.hyperhive.swarm.hives`, the same directory every host holds. See
[swarm/](../../2-handbook/swarm/README.md) for the mesh itself.
Note that `deploy.wireguard.enable` gates the mesh, **not**
`deploy.hive-controller.enable` --- a store host runs no hive and would otherwise get no
`wg-hive` interface at all.
## Pointing a hive at it
The block above configures the host that _receives_. You must tell every
hive that _pushes_ separately where the store is:
```nix
services.hyperhive.swarm.snapshotStore = {
address = "10.100.0.9"; # the store's mesh address, no prefix
port = 51821; # optional; must match the receiver's port
};
```
Two deliberate asymmetries in that pair, both liable to be misread as
inconsistency:
- **`address` has no default.** It's a deployment fact a pushing hive
can't derive, and a wrong guess means streaming an agent's state at
whatever happens to answer. Unset, a push fails naming this option.
- **`port` does default** (`51821`), because it's a convention both
ends read from the same option docs --- a default there is
coordination, not a guess.
Note the option lives under `swarm.*` while the receiving host's lives
under `services.hyperhive.snapshotStore`. That's the distinction the
two namespaces carry throughout: `swarm.*` describes _the swarm_ as seen
from this host, and a bare `services.hyperhive.<service>` describes _a
role this host performs_. A store host sets both --- one to run the
receiver, one only if it also runs a hive that pushes.
With it set, `hivectl agent <name> subvol snapshot push <label>
[--parent <label>]` streams a snapshot straight into the store. No
destination argument exists, because a swarm has exactly one store (see
[One subvolume per agent, not per hive](#one-subvolume-per-agent-not-per-hive)),
and no credential argument, because the mesh is the authentication.
## The mesh is the authentication
No certificates live here, and no key material of its own. That's
deliberate rather than an omission.
WireGuard's cryptokey routing already binds a peer's source address to
its public key: the swarm module configures each peer with
`allowedIPs = [ peer.wireguardAddress ]`, so a packet arriving from
that address provably came from the holder of that private key. By the
time a packet reaches the receiver, the kernel has already
authenticated it.
Layering TLS client certs on top would authenticate _the same fact_ a
second time, and add a credential with an expiry --- a migration that
fails because a renewal quietly didn't happen, discovered on the day
you need to move an agent.
## One subvolume per agent, not per hive
The receiver keys the destination by **agent**.
This isn't cosmetic. After a migration, an agent's next incremental
send arrives from a _different_ hive than the previous one. Keying by
hive would split that agent's snapshot chain across two directories,
and `btrfs send -p` would fail to find its parent --- breaking exactly
the case the store exists to serve.
## What the sender can and can't choose
A `btrfs send` stream carries no notion of _which agent_ it belongs to,
and the subvolume name inside it's chosen by the sender. The
protocol is one `agent <name>` header line, then the raw stream.
The receiver waits 30 s for the header, and systemd stops a transfer
that runs past 6 h in total. The sender gives connect and header write 15 s
each; it puts no limit on the stream itself.
The rule that matters:
> **The receiver owns the destination root. The sender-supplied name is
> validated, never used as a path.**
Validation is an allowlist --- `[A-Za-z0-9_-]+` and nothing else. No
slash and no dot means neither directory traversal nor an absolute path
can survive it. It's deliberately an allowlist and not a list of
forbidden characters: a blocklist only ever excludes the attacks
somebody already thought of.
## Reachability
The receiver is socket-activated, and the socket binds **this host's
mesh address**, never a wildcard. The module asserts three things: the
mesh is on, the address isn't empty, and it isn't `0.0.0.0` or `::` ---
none is just documentation: bound to a wildcard this socket is an
unauthenticated remote write into agent state.
The socket sets `FreeBind`, so the bind succeeds before `wg-hive` has
its address. Nothing orders the wireguard unit before `sockets.target`,
and without it the bind fails with `EADDRNOTAVAIL` at boot and the
receiver never listens.
Binding isn't sufficient on its own. NixOS's firewall is default-deny
and filters in netfilter, _before_ a packet reaches a bound socket, so
the module explicitly opens the port --- scoped to the mesh interface:
```nix
networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ cfg.port ];
```
A host-wide `allowedTCPPorts` would open the port on every interface
including a public NIC, leaving only the socket's bind address between
the internet and a root `btrfs receive`.
## Operating it
### Confinement is the deployment's job
`btrfs receive` needs `CAP_SYS_ADMIN`, so the receiver runs as root.
The unit sets `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp` and a
narrow `ReadWritePaths` --- but those are **defence in depth, not a
boundary**: a process holding `CAP_SYS_ADMIN` can call `mount(2)` and
undo the namespace they set up.
The boundary is the machine. The intended deployments are:
- **a swarm**: the store is its own small VM. The machine is the
boundary, which is stronger than anything the unit could assert about
itself.
- **all-in-one / local**: the store runs as a container on the c0re
host.
The second is worth keeping deliberately, and not only for
convenience: it means every local deployment exercises the confined
path. The usual failure mode for an isolated variant is that
nobody runs it day to day, so it rots and turns up broken in
production.
⚠️ **The assumption to keep true over time:** the store host runs
nothing else. That's true on day one and quietly false the day someone
notices the box has spare disk. Nothing in the config objects when it
stops being true.
### It holds every agent's state from every hive
Which makes it the highest-value target in the swarm by a wide margin,
and means it should get the treatment a backup host gets --- restricted
access, and a decision (rather than an omission) on encryption at rest.
The pitfall is the label: this box holds backup-grade data while nobody
calls it a backup, so it can end up with backup-grade _exposure_ and
non-backup-grade _controls_. Nobody puts a migration staging area on
the access-review list.
### What a snapshot contains
The snapshot covers an agent's **state subvolume**, which is the parent
of `state/`, `claude/` and `harness/` (see
[`docs/2-handbook/agents/persistence.md`'s btrfs subvolume
section](../../2-handbook/agents/persistence.md#btrfs-subvolumes-for-varlibhyperhiveagentsname)
for how and when hive-c0re creates that subvolume). Consequences:
<!-- vale write-good.Passive = NO -->
- The Claude session (`claude/`) travels, so a restored agent keeps its
live `--continue` session rather than needing to log in again.
- `harness/` travels too, including `harness/bash-tasks/`. Task output
is part of an agent's working continuity, so this is wanted --- but it
means anything that has ever leaked into a task's captured output is
in the retained snapshots as well.
<!-- vale write-good.Passive = YES -->
It does **not** cover the agent's config (the forge's `agent-configs/<name>`) or
its topology entry, both of which live outside the subvolume. A restore
therefore yields an agent's memory without its definition.
### Retention
<!-- vale write-good.Passive = NO -->
Retention lives on the _sending_ side (last-N by count, swept
periodically), not here. Count rather than age is deliberate: a count
is bounded by construction, whereas an age policy silently scales disk
usage with how hot a hive runs.
<!-- vale write-good.Passive = YES -->
Per-agent or per-hive `btrfs qgroup` quotas aren't configured yet.
Without them one runaway hive can fill the store and take out every
other hive's snapshots.
## Not built yet
**The pull side.** Push is safe with minimal authorisation because a
hive can only ever write to a chain it owns. Pull is the direction that
needs a policy: unrestricted, any compromised hive could read every
agent's state from every other hive. It needs a notion of which hive
currently owns which agent, and that ownership record lands with the
swarm controller work.
With a single hive the question is trivial --- the only peer owns
everything it sends --- which is why the receive half ships first.

2
docs/4-implementation/trust-boundary/hive-c0re-privilege.md
View file

@ -57,7 +57,7 @@ known operations; there is no arbitrary command pass-through:
| `SnapshotAgentSubvolume` | `btrfs subvolume snapshot -r <agent_root> <snapshot_path>` — the frozen point-in-time copy a migration streams from |
| `DeleteAgentSnapshot` | `btrfs subvolume delete <snapshot_path>` |
| `SendAgentSnapshotToFile` | `btrfs send [-p <parent>] <snapshot>` into a bare filename under `MIGRATE_STAGING_ROOT`; refuses to overwrite an existing export |
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). hive-c0re connects to the peer and hands over the connected socket, so hive-priv never learns an address or protocol. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
| `SendAgentSnapshotToFd` | `btrfs send [-p <parent>] <snapshot>` into a **file descriptor passed with the request** (`SCM_RIGHTS`). The caller connects to the destination and hands over the connected socket, so hive-priv never learns an address or protocol; hive-c0re sends no such request today. hive-priv requires exactly one descriptor here, and refuses one that arrives alongside any other operation |
**Strings that reach root-loaded config** — `WriteResourceLimits` and
`WriteNspawnFlags` copy request strings into a systemd drop-in and an env