vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(#2862): document the snapshot store, drop the dedicated option

Browse source 
mara: the option was the wrong shape for the concern. "this host runs
nothing else" is a deployment expectation, not something a module
should assert about its own host — and asserting it made co-location
look like a config toggle rather than what it is.

Replaced with docs/snapshot-store.md, which the module had no docs
page at all before: enabling it, why the mesh is the authentication
(cryptokey routing already binds source address to pubkey, so certs
would authenticate the same fact twice and add an expiry), why the
destination is keyed per agent (a per-hive prefix splits an agent's
chain the first time it migrates), what the sender may and may not
choose, why the firewall rule is interface-scoped, what a snapshot
does and does not contain, and what the pull side still needs.

The dedicated-host expectation is stated there as an operational
assumption with its own failure mode — true on day one, quietly false
the day someone notices the box has spare disk — rather than as an
assertion someone flips to false to make the build proceed.

Linked from CLAUDE.md's reading paths.
This commit is contained in:
atlas 2026-07-31 18:34:34 +02:00 • committed by mara
commit c051cd9717
3 changed files with 193 additions and 33 deletions
Download patch file Download diff file Expand all files Collapse all files

4
CLAUDE.md
View file

@ -165,6 +165,10 @@ read them à la carte.
- **"How do I connect two hives into a swarm? How do I declare peer
hives and configure TLS trust?"** →
[`docs/swarm.md`](docs/swarm.md).
- **"Where do agent snapshots go? How does the swarm's `btrfs receive`
endpoint authenticate a pushing hive, and what does a snapshot
actually contain?"** →
[`docs/snapshot-store.md`](docs/snapshot-store.md).
- **"How does the rebuild queue work? What are queue kinds and sources?"** →
[`docs/coordinator.md`](docs/coordinator.md).
- **"How does the CI runner work? What's the auto-registration flow?"** →

186
docs/snapshot-store.md Normal file
View file

@ -0,0 +1,186 @@
# Snapshot store
The swarm's `btrfs receive` endpoint. Hives push agent snapshots to it
over the WireGuard mesh; a destination hive later pulls one back to
complete a migration.
Two things it is not, both worth stating because both are easy to
assume:
- **It is not the swarm controller**, and does not depend on one. It is
a NixOS host role: a btrfs subvolume tree, a socket-activated
receiver, and the `wg-hive` interface the swarm module already brings
up. That is why it can be deployed before any controller exists.
- **It is not a backup product.** It happens to hold the data a backup
would hold, and it should be operated accordingly (see
[Operating it](#operating-it)) --- but nothing in it does scheduling,
verification, or restore orchestration.
## Enabling it
```nix
services.hyperhive.snapshotStore = {
enable = true;
path = "/var/lib/hyperhive-snapshots"; # must be on btrfs
port = 51821;
};
# The mesh is a hard requirement, and is asserted:
services.hyperhive.swarm.wireguard = {
enable = true;
address = "10.100.0.9/24";
privateKeyFile = "/etc/wireguard/hive.key";
};
```
The store host is a swarm member like any other: peers declare it, and
it declares them, through `services.hyperhive.swarm.peers`. See
[swarm.md](swarm.md) for the mesh itself.
Note that the mesh is gated on `swarm.wireguard.enable`, **not** on
`c0re.enable` --- a store host runs no hive and would otherwise get no
`wg-hive` interface at all.
## The mesh is the authentication
There are no certificates here, and no key material of its own. That is
deliberate rather than an omission.
WireGuard's cryptokey routing already binds a peer's source address to
its public key: the swarm module configures each peer with
`allowedIPs = [ peer.wireguardAddress ]`, so a packet arriving from
that address provably came from the holder of that private key. A
packet that reaches the receiver has therefore already been
authenticated by the kernel.
Layering TLS client certs on top would authenticate *the same fact* a
second time, and add a credential with an expiry --- a migration that
fails because a renewal quietly didn't happen, discovered on the day
you need to move an agent.
## One subvolume per agent, not per hive
The destination is keyed by **agent**.
This is not cosmetic. After a migration, an agent's next incremental
send arrives from a *different* hive than the previous one. Keying by
hive would split that agent's snapshot chain across two directories,
and `btrfs send -p` would fail to find its parent --- breaking exactly
the case the store exists to serve.
## What the sender can and cannot choose
A `btrfs send` stream carries no notion of *which agent* it belongs to,
and the subvolume name inside it is chosen by the sender. So the
protocol is one `agent <name>` header line, then the raw stream.
The rule that matters:
> **The receiver owns the destination root. The sender-supplied name is
> validated, never used as a path.**
Validation is a whitelist --- `[A-Za-z0-9_-]+` and nothing else. No
slash and no dot means neither directory traversal nor an absolute path
can survive it. It is deliberately a whitelist and not a list of
forbidden characters: a blocklist only ever excludes the attacks
somebody already thought of.
## Reachability
The receiver is socket-activated, and the socket binds **this host's
mesh address**, never a wildcard. Both the mesh being enabled and the
address being set are assertions, not documentation --- bound to
`0.0.0.0` this socket is an unauthenticated remote write into agent
state.
Binding is not sufficient on its own. NixOS's firewall is default-deny
and filters in netfilter, *before* a packet reaches a bound socket, so
the port is opened explicitly --- and scoped to the mesh interface:
```nix
networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ cfg.port ];
```
A host-wide `allowedTCPPorts` would open the port on every interface
including a public NIC, leaving only the socket's bind address between
the internet and a root `btrfs receive`.
## Operating it
### Confinement is the deployment's job
`btrfs receive` needs `CAP_SYS_ADMIN`, so the receiver runs as root.
The unit sets `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp` and a
narrow `ReadWritePaths` --- but those are **defence in depth, not a
boundary**: a process holding `CAP_SYS_ADMIN` can call `mount(2)` and
undo the namespace they set up.
The boundary is the machine. The intended deployments are:
- **a swarm**: the store is its own small VM. The machine is the
boundary, which is stronger than anything the unit could assert about
itself.
- **all-in-one / local**: the store runs as a container on the c0re
host.
The second is worth keeping deliberately, and not only for
convenience: it means the confined path is exercised by every local
deployment. The usual failure mode for an isolated variant is that
nobody runs it day to day, so it rots and is discovered broken in
production.
⚠️ **The assumption to keep true over time:** the store host runs
nothing else. That is true on day one and quietly false the day someone
notices the box has spare disk. Nothing in the config objects when it
stops being true.
### It holds every agent's state from every hive
Which makes it the highest-value target in the swarm by a wide margin,
and means it should get the treatment a backup host gets --- restricted
access, and a decision (rather than an omission) on encryption at rest.
The trap is the label: this box holds backup-grade data while not being
called a backup, so it can end up with backup-grade *exposure* and
non-backup-grade *controls*. Nobody puts a migration staging area on
the access-review list.
### What a snapshot contains
The snapshot covers an agent's **state subvolume**, which is the parent
of `state/`, `claude/` and `harness/`. Consequences:
- The Claude session (`claude/`) travels, so a restored agent keeps its
live `--continue` session rather than needing to log in again.
- `harness/` travels too, including `harness/bash-tasks/`. Task output
is part of an agent's working continuity, so this is wanted --- but it
means anything that has ever leaked into a task's captured output is
in the retained snapshots as well.
It does **not** cover the agent's applied config (`/applied/<name>/`) or
its topology entry, both of which live outside the subvolume. A restore
therefore yields an agent's memory without its definition; closing that
gap is tracked separately.
### Retention
Retention lives on the *sending* side (last-N by count, swept
periodically), not here. Count rather than age is deliberate: a count
is bounded by construction, whereas an age policy silently scales disk
usage with how hot a hive runs.
Per-agent or per-hive `btrfs qgroup` quotas are not configured yet.
Without them one runaway hive can fill the store and take out every
other hive's snapshots.
## Not built yet
**The pull side.** Push is safe with minimal authorisation because a
hive can only ever write to a chain it owns. Pull is the direction that
needs a policy: unrestricted, any compromised hive could read every
agent's state from every other hive. It needs a notion of which hive
currently owns which agent, and that ownership record lands with the
swarm controller work.
With a single hive the question is trivial --- the only peer owns
everything it sends --- which is why the receive half ships first.

36
nix/host-modules/hive-snapshot-store.nix
View file

@ -16,7 +16,8 @@
# Confinement is a property of the DEPLOYMENT, not of this unit: in a
# real swarm the store is its own small VM (the machine is the
# boundary); in the all-local case it's a container on the c0re host.
# The module therefore hardcodes neither --- see `dedicated` below.
# The module hardcodes neither. docs/snapshot-store.md covers what the
# deployment is expected to provide.
{
pkgs,
lib,
@ -129,23 +130,6 @@ in
'';
};
dedicated = lib.mkOption {
type = lib.types.bool;
default = true;
description = ''
Assert that this host runs no other hyperhive role. The store
aggregates every agent's state from every hive in the swarm, so
the intended deployment is a dedicated machine (or a container
in the all-local case) where the machine itself is the security
boundary.
That assumption is true on day one and silently false the day
someone notices the box has spare disk. This option makes it a
thing the build checks rather than a thing the deployer
remembers. Set to `false` to co-locate deliberately --- the
point is that it becomes a decision, not an accident.
'';
};
};
config = lib.mkIf cfg.enable {
@ -170,20 +154,6 @@ in
fall back to a wildcard.
'';
}
{
assertion = !cfg.dedicated || !config.services.hyperhive.c0re.enable;
message = ''
services.hyperhive.snapshotStore is enabled alongside
services.hyperhive.c0re on the same host. The store holds
every agent's state from every hive, so it is meant to run on
a machine of its own where the machine is the boundary.
If the co-location is deliberate (the all-local single-host
deployment, where the store runs as a container), set
services.hyperhive.snapshotStore.dedicated = false to record
that decision explicitly.
'';
}
];
# The store root must exist before the first connection arrives ---
@ -231,7 +201,7 @@ in
# nature. The hardening below is defence in depth and NOT a
# boundary: a process holding CAP_SYS_ADMIN can call mount(2) and
# undo the namespace these directives set up. The real boundary is
# the deployment (dedicated host / container) --- see `dedicated`.
# the deployment --- see docs/snapshot-store.md.
systemd.services."hive-snapshot-store@" = {
description = "hyperhive swarm snapshot store receiver";
after = [ "hive-snapshot-store.socket" ];