224 lines
9 KiB
Markdown
224 lines
9 KiB
Markdown
# Snapshot store
|
|
|
|
The swarm's `btrfs receive` endpoint. Hives push agent snapshots to it
|
|
over the WireGuard mesh; a destination hive later pulls one back to
|
|
complete a migration.
|
|
|
|
Two things it is not, both worth stating because both are easy to
|
|
assume:
|
|
|
|
- **It is not the swarm controller**, and does not depend on one. It is
|
|
a NixOS host role: a btrfs subvolume tree, a socket-activated
|
|
receiver, and the `wg-hive` interface the swarm module already brings
|
|
up. That is why it can be deployed before any controller exists.
|
|
- **It is not a backup product.** It happens to hold the data a backup
|
|
would hold, and it should be operated accordingly (see
|
|
[Operating it](#operating-it)) --- but nothing in it does scheduling,
|
|
verification, or restore orchestration.
|
|
|
|
## Enabling it
|
|
|
|
```nix
|
|
services.hyperhive.snapshotStore = {
|
|
enable = true;
|
|
path = "/var/lib/hyperhive-snapshots"; # must be on btrfs
|
|
port = 51821;
|
|
};
|
|
|
|
# The mesh is a hard requirement, and is asserted:
|
|
services.hyperhive.swarm.wireguard = {
|
|
enable = true;
|
|
address = "10.100.0.9/24";
|
|
privateKeyFile = "/etc/wireguard/hive.key";
|
|
};
|
|
```
|
|
|
|
The store host is a swarm member like any other: it gets an entry in
|
|
`services.hyperhive.swarm.hives`, the same directory every host holds. See
|
|
[swarm/](swarm/README.md) for the mesh itself.
|
|
|
|
Note that the mesh is gated on `swarm.wireguard.enable`, **not** on
|
|
`c0re.enable` --- a store host runs no hive and would otherwise get no
|
|
`wg-hive` interface at all.
|
|
|
|
## Pointing a hive at it
|
|
|
|
The block above configures the host that *receives*. Every hive that
|
|
*pushes* separately needs to be told where the store is:
|
|
|
|
```nix
|
|
services.hyperhive.swarm.snapshotStore = {
|
|
address = "10.100.0.9"; # the store's mesh address, no prefix
|
|
port = 51821; # optional; must match the receiver's port
|
|
};
|
|
```
|
|
|
|
Two deliberate asymmetries in that pair, both easy to misread as
|
|
inconsistency:
|
|
|
|
- **`address` has no default.** It is a deployment fact a pushing hive
|
|
cannot derive, and a wrong guess means streaming an agent's state at
|
|
whatever happens to answer. Unset, a push fails naming this option.
|
|
- **`port` does default** (`51821`), because it is a convention both
|
|
ends read from the same option docs --- a default there is
|
|
coordination, not a guess.
|
|
|
|
Note the option lives under `swarm.*` while the receiving host's lives
|
|
under `services.hyperhive.snapshotStore`. That is the distinction the
|
|
two namespaces carry throughout: `swarm.*` describes *the swarm* as seen
|
|
from this host, and a bare `services.hyperhive.<service>` describes *a
|
|
role this host performs*. A store host sets both --- one to run the
|
|
receiver, one only if it also runs a hive that pushes.
|
|
|
|
With it set, `hivectl agent <name> subvol snapshot push <label>
|
|
[--parent <label>]` streams a snapshot straight into the store. There is
|
|
no destination argument, because a swarm has exactly one store (see
|
|
[One subvolume per agent, not per hive](#one-subvolume-per-agent-not-per-hive)),
|
|
and no credential argument, because the mesh is the authentication.
|
|
|
|
## The mesh is the authentication
|
|
|
|
There are no certificates here, and no key material of its own. That is
|
|
deliberate rather than an omission.
|
|
|
|
WireGuard's cryptokey routing already binds a peer's source address to
|
|
its public key: the swarm module configures each peer with
|
|
`allowedIPs = [ peer.wireguardAddress ]`, so a packet arriving from
|
|
that address provably came from the holder of that private key. A
|
|
packet that reaches the receiver has therefore already been
|
|
authenticated by the kernel.
|
|
|
|
Layering TLS client certs on top would authenticate *the same fact* a
|
|
second time, and add a credential with an expiry --- a migration that
|
|
fails because a renewal quietly didn't happen, discovered on the day
|
|
you need to move an agent.
|
|
|
|
## One subvolume per agent, not per hive
|
|
|
|
The destination is keyed by **agent**.
|
|
|
|
This is not cosmetic. After a migration, an agent's next incremental
|
|
send arrives from a *different* hive than the previous one. Keying by
|
|
hive would split that agent's snapshot chain across two directories,
|
|
and `btrfs send -p` would fail to find its parent --- breaking exactly
|
|
the case the store exists to serve.
|
|
|
|
## What the sender can and cannot choose
|
|
|
|
A `btrfs send` stream carries no notion of *which agent* it belongs to,
|
|
and the subvolume name inside it is chosen by the sender. So the
|
|
protocol is one `agent <name>` header line, then the raw stream.
|
|
|
|
The rule that matters:
|
|
|
|
> **The receiver owns the destination root. The sender-supplied name is
|
|
> validated, never used as a path.**
|
|
|
|
Validation is a whitelist --- `[A-Za-z0-9_-]+` and nothing else. No
|
|
slash and no dot means neither directory traversal nor an absolute path
|
|
can survive it. It is deliberately a whitelist and not a list of
|
|
forbidden characters: a blocklist only ever excludes the attacks
|
|
somebody already thought of.
|
|
|
|
## Reachability
|
|
|
|
The receiver is socket-activated, and the socket binds **this host's
|
|
mesh address**, never a wildcard. Both the mesh being enabled and the
|
|
address being set are assertions, not documentation --- bound to
|
|
`0.0.0.0` this socket is an unauthenticated remote write into agent
|
|
state.
|
|
|
|
Binding is not sufficient on its own. NixOS's firewall is default-deny
|
|
and filters in netfilter, *before* a packet reaches a bound socket, so
|
|
the port is opened explicitly --- and scoped to the mesh interface:
|
|
|
|
```nix
|
|
networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ cfg.port ];
|
|
```
|
|
|
|
A host-wide `allowedTCPPorts` would open the port on every interface
|
|
including a public NIC, leaving only the socket's bind address between
|
|
the internet and a root `btrfs receive`.
|
|
|
|
## Operating it
|
|
|
|
### Confinement is the deployment's job
|
|
|
|
`btrfs receive` needs `CAP_SYS_ADMIN`, so the receiver runs as root.
|
|
The unit sets `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp` and a
|
|
narrow `ReadWritePaths` --- but those are **defence in depth, not a
|
|
boundary**: a process holding `CAP_SYS_ADMIN` can call `mount(2)` and
|
|
undo the namespace they set up.
|
|
|
|
The boundary is the machine. The intended deployments are:
|
|
|
|
- **a swarm**: the store is its own small VM. The machine is the
|
|
boundary, which is stronger than anything the unit could assert about
|
|
itself.
|
|
- **all-in-one / local**: the store runs as a container on the c0re
|
|
host.
|
|
|
|
The second is worth keeping deliberately, and not only for
|
|
convenience: it means the confined path is exercised by every local
|
|
deployment. The usual failure mode for an isolated variant is that
|
|
nobody runs it day to day, so it rots and is discovered broken in
|
|
production.
|
|
|
|
⚠️ **The assumption to keep true over time:** the store host runs
|
|
nothing else. That is true on day one and quietly false the day someone
|
|
notices the box has spare disk. Nothing in the config objects when it
|
|
stops being true.
|
|
|
|
### It holds every agent's state from every hive
|
|
|
|
Which makes it the highest-value target in the swarm by a wide margin,
|
|
and means it should get the treatment a backup host gets --- restricted
|
|
access, and a decision (rather than an omission) on encryption at rest.
|
|
|
|
The trap is the label: this box holds backup-grade data while not being
|
|
called a backup, so it can end up with backup-grade *exposure* and
|
|
non-backup-grade *controls*. Nobody puts a migration staging area on
|
|
the access-review list.
|
|
|
|
### What a snapshot contains
|
|
|
|
The snapshot covers an agent's **state subvolume**, which is the parent
|
|
of `state/`, `claude/` and `harness/` (see
|
|
[`docs/persistence.md`'s btrfs subvolume
|
|
section](persistence.md#btrfs-subvolumes-for-varlibhyperhiveagentsname)
|
|
for how and when that subvolume is created). Consequences:
|
|
|
|
- The Claude session (`claude/`) travels, so a restored agent keeps its
|
|
live `--continue` session rather than needing to log in again.
|
|
- `harness/` travels too, including `harness/bash-tasks/`. Task output
|
|
is part of an agent's working continuity, so this is wanted --- but it
|
|
means anything that has ever leaked into a task's captured output is
|
|
in the retained snapshots as well.
|
|
|
|
It does **not** cover the agent's applied config (`/applied/<name>/`) or
|
|
its topology entry, both of which live outside the subvolume. A restore
|
|
therefore yields an agent's memory without its definition; closing that
|
|
gap is tracked separately.
|
|
|
|
### Retention
|
|
|
|
Retention lives on the *sending* side (last-N by count, swept
|
|
periodically), not here. Count rather than age is deliberate: a count
|
|
is bounded by construction, whereas an age policy silently scales disk
|
|
usage with how hot a hive runs.
|
|
|
|
Per-agent or per-hive `btrfs qgroup` quotas are not configured yet.
|
|
Without them one runaway hive can fill the store and take out every
|
|
other hive's snapshots.
|
|
|
|
## Not built yet
|
|
|
|
**The pull side.** Push is safe with minimal authorisation because a
|
|
hive can only ever write to a chain it owns. Pull is the direction that
|
|
needs a policy: unrestricted, any compromised hive could read every
|
|
agent's state from every other hive. It needs a notion of which hive
|
|
currently owns which agent, and that ownership record lands with the
|
|
swarm controller work.
|
|
|
|
With a single hive the question is trivial --- the only peer owns
|
|
everything it sends --- which is why the receive half ships first.
|