diff --git a/CLAUDE.md b/CLAUDE.md index 1c72fa66..9d811e00 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -165,6 +165,10 @@ read them à la carte. - **"How do I connect two hives into a swarm? How do I declare peer hives and configure TLS trust?"** → [`docs/swarm.md`](docs/swarm.md). +- **"Where do agent snapshots go? How does the swarm's `btrfs receive` + endpoint authenticate a pushing hive, and what does a snapshot + actually contain?"** → + [`docs/snapshot-store.md`](docs/snapshot-store.md). - **"How does the rebuild queue work? What are queue kinds and sources?"** → [`docs/coordinator.md`](docs/coordinator.md). - **"How does the CI runner work? What's the auto-registration flow?"** → diff --git a/docs/snapshot-store.md b/docs/snapshot-store.md new file mode 100644 index 00000000..8bacdd21 --- /dev/null +++ b/docs/snapshot-store.md @@ -0,0 +1,186 @@ +# Snapshot store + +The swarm's `btrfs receive` endpoint. Hives push agent snapshots to it +over the WireGuard mesh; a destination hive later pulls one back to +complete a migration. + +Two things it is not, both worth stating because both are easy to +assume: + +- **It is not the swarm controller**, and does not depend on one. It is + a NixOS host role: a btrfs subvolume tree, a socket-activated + receiver, and the `wg-hive` interface the swarm module already brings + up. That is why it can be deployed before any controller exists. +- **It is not a backup product.** It happens to hold the data a backup + would hold, and it should be operated accordingly (see + [Operating it](#operating-it)) --- but nothing in it does scheduling, + verification, or restore orchestration. + +## Enabling it + +```nix +services.hyperhive.snapshotStore = { + enable = true; + path = "/var/lib/hyperhive-snapshots"; # must be on btrfs + port = 51821; +}; + +# The mesh is a hard requirement, and is asserted: +services.hyperhive.swarm.wireguard = { + enable = true; + address = "10.100.0.9/24"; + privateKeyFile = "/etc/wireguard/hive.key"; +}; +``` + +The store host is a swarm member like any other: peers declare it, and +it declares them, through `services.hyperhive.swarm.peers`. See +[swarm.md](swarm.md) for the mesh itself. + +Note that the mesh is gated on `swarm.wireguard.enable`, **not** on +`c0re.enable` --- a store host runs no hive and would otherwise get no +`wg-hive` interface at all. + +## The mesh is the authentication + +There are no certificates here, and no key material of its own. That is +deliberate rather than an omission. + +WireGuard's cryptokey routing already binds a peer's source address to +its public key: the swarm module configures each peer with +`allowedIPs = [ peer.wireguardAddress ]`, so a packet arriving from +that address provably came from the holder of that private key. A +packet that reaches the receiver has therefore already been +authenticated by the kernel. + +Layering TLS client certs on top would authenticate *the same fact* a +second time, and add a credential with an expiry --- a migration that +fails because a renewal quietly didn't happen, discovered on the day +you need to move an agent. + +## One subvolume per agent, not per hive + +The destination is keyed by **agent**. + +This is not cosmetic. After a migration, an agent's next incremental +send arrives from a *different* hive than the previous one. Keying by +hive would split that agent's snapshot chain across two directories, +and `btrfs send -p` would fail to find its parent --- breaking exactly +the case the store exists to serve. + +## What the sender can and cannot choose + +A `btrfs send` stream carries no notion of *which agent* it belongs to, +and the subvolume name inside it is chosen by the sender. So the +protocol is one `agent ` header line, then the raw stream. + +The rule that matters: + +> **The receiver owns the destination root. The sender-supplied name is +> validated, never used as a path.** + +Validation is a whitelist --- `[A-Za-z0-9_-]+` and nothing else. No +slash and no dot means neither directory traversal nor an absolute path +can survive it. It is deliberately a whitelist and not a list of +forbidden characters: a blocklist only ever excludes the attacks +somebody already thought of. + +## Reachability + +The receiver is socket-activated, and the socket binds **this host's +mesh address**, never a wildcard. Both the mesh being enabled and the +address being set are assertions, not documentation --- bound to +`0.0.0.0` this socket is an unauthenticated remote write into agent +state. + +Binding is not sufficient on its own. NixOS's firewall is default-deny +and filters in netfilter, *before* a packet reaches a bound socket, so +the port is opened explicitly --- and scoped to the mesh interface: + +```nix +networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ cfg.port ]; +``` + +A host-wide `allowedTCPPorts` would open the port on every interface +including a public NIC, leaving only the socket's bind address between +the internet and a root `btrfs receive`. + +## Operating it + +### Confinement is the deployment's job + +`btrfs receive` needs `CAP_SYS_ADMIN`, so the receiver runs as root. +The unit sets `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp` and a +narrow `ReadWritePaths` --- but those are **defence in depth, not a +boundary**: a process holding `CAP_SYS_ADMIN` can call `mount(2)` and +undo the namespace they set up. + +The boundary is the machine. The intended deployments are: + +- **a swarm**: the store is its own small VM. The machine is the + boundary, which is stronger than anything the unit could assert about + itself. +- **all-in-one / local**: the store runs as a container on the c0re + host. + +The second is worth keeping deliberately, and not only for +convenience: it means the confined path is exercised by every local +deployment. The usual failure mode for an isolated variant is that +nobody runs it day to day, so it rots and is discovered broken in +production. + +⚠️ **The assumption to keep true over time:** the store host runs +nothing else. That is true on day one and quietly false the day someone +notices the box has spare disk. Nothing in the config objects when it +stops being true. + +### It holds every agent's state from every hive + +Which makes it the highest-value target in the swarm by a wide margin, +and means it should get the treatment a backup host gets --- restricted +access, and a decision (rather than an omission) on encryption at rest. + +The trap is the label: this box holds backup-grade data while not being +called a backup, so it can end up with backup-grade *exposure* and +non-backup-grade *controls*. Nobody puts a migration staging area on +the access-review list. + +### What a snapshot contains + +The snapshot covers an agent's **state subvolume**, which is the parent +of `state/`, `claude/` and `harness/`. Consequences: + +- The Claude session (`claude/`) travels, so a restored agent keeps its + live `--continue` session rather than needing to log in again. +- `harness/` travels too, including `harness/bash-tasks/`. Task output + is part of an agent's working continuity, so this is wanted --- but it + means anything that has ever leaked into a task's captured output is + in the retained snapshots as well. + +It does **not** cover the agent's applied config (`/applied//`) or +its topology entry, both of which live outside the subvolume. A restore +therefore yields an agent's memory without its definition; closing that +gap is tracked separately. + +### Retention + +Retention lives on the *sending* side (last-N by count, swept +periodically), not here. Count rather than age is deliberate: a count +is bounded by construction, whereas an age policy silently scales disk +usage with how hot a hive runs. + +Per-agent or per-hive `btrfs qgroup` quotas are not configured yet. +Without them one runaway hive can fill the store and take out every +other hive's snapshots. + +## Not built yet + +**The pull side.** Push is safe with minimal authorisation because a +hive can only ever write to a chain it owns. Pull is the direction that +needs a policy: unrestricted, any compromised hive could read every +agent's state from every other hive. It needs a notion of which hive +currently owns which agent, and that ownership record lands with the +swarm controller work. + +With a single hive the question is trivial --- the only peer owns +everything it sends --- which is why the receive half ships first. diff --git a/nix/host-modules/hive-snapshot-store.nix b/nix/host-modules/hive-snapshot-store.nix index 2a31c5f8..5ffa3cbf 100644 --- a/nix/host-modules/hive-snapshot-store.nix +++ b/nix/host-modules/hive-snapshot-store.nix @@ -16,7 +16,8 @@ # Confinement is a property of the DEPLOYMENT, not of this unit: in a # real swarm the store is its own small VM (the machine is the # boundary); in the all-local case it's a container on the c0re host. -# The module therefore hardcodes neither --- see `dedicated` below. +# The module hardcodes neither. docs/snapshot-store.md covers what the +# deployment is expected to provide. { pkgs, lib, @@ -129,23 +130,6 @@ in ''; }; - dedicated = lib.mkOption { - type = lib.types.bool; - default = true; - description = '' - Assert that this host runs no other hyperhive role. The store - aggregates every agent's state from every hive in the swarm, so - the intended deployment is a dedicated machine (or a container - in the all-local case) where the machine itself is the security - boundary. - - That assumption is true on day one and silently false the day - someone notices the box has spare disk. This option makes it a - thing the build checks rather than a thing the deployer - remembers. Set to `false` to co-locate deliberately --- the - point is that it becomes a decision, not an accident. - ''; - }; }; config = lib.mkIf cfg.enable { @@ -170,20 +154,6 @@ in fall back to a wildcard. ''; } - { - assertion = !cfg.dedicated || !config.services.hyperhive.c0re.enable; - message = '' - services.hyperhive.snapshotStore is enabled alongside - services.hyperhive.c0re on the same host. The store holds - every agent's state from every hive, so it is meant to run on - a machine of its own where the machine is the boundary. - - If the co-location is deliberate (the all-local single-host - deployment, where the store runs as a container), set - services.hyperhive.snapshotStore.dedicated = false to record - that decision explicitly. - ''; - } ]; # The store root must exist before the first connection arrives --- @@ -231,7 +201,7 @@ in # nature. The hardening below is defence in depth and NOT a # boundary: a process holding CAP_SYS_ADMIN can call mount(2) and # undo the namespace these directives set up. The real boundary is - # the deployment (dedicated host / container) --- see `dedicated`. + # the deployment --- see docs/snapshot-store.md. systemd.services."hive-snapshot-store@" = { description = "hyperhive swarm snapshot store receiver"; after = [ "hive-snapshot-store.socket" ];