mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-archive: external s3 backend (options, prune path, docs, module-eval cases, vm fake s3)

Browse source 
- swarm-archive.nix, lib/archive-server.nix: deploy.archive.backend and
  deploy.archive.s3.* options; the s3 server and the prune read the key from the
  store into rclone's environment; the prune lists and addresses repositories
  through rclone instead of a local directory
- swarm-bao.nix: the archive reads swarm/services/archive/s3/*
- docs: archive.md "External S3" section, two credentials-reference rows
- module-eval: builder.nix cases for local and s3, the missing-setting
  assertion; bao-grants.nix gains the s3 read grant
- vm test: second node with a fake rclone serve s3 whose key is generated at
  test time
This commit is contained in:
atlas 2026-11-10 20:47:49 +01:00 • committed by mara
commit 86c5dac89d
1 changed files with 605 additions and 156 deletions
Download patch file Download diff file Expand all files Collapse all files

61
docs/2-handbook/swarm/archive.md
View file

@ -33,8 +33,9 @@ as for the binary cache.
Each agent has its own [restic](https://restic.net) repository at
`https://archive.<swarm domain>/<agent>/`, under `deploy.archive.dataDir`
(default `/var/lib/swarm-archive`) on that host. restic encrypts it with a
password of its own for that agent.
(default `/var/lib/swarm-archive`) on that host, or in a bucket of an
[external S3](#external-s3). restic encrypts it with a password of its own for
that agent.
Two properties protect the backups from a compromised hive:
@ -49,6 +50,54 @@ Two properties protect the backups from a compromised hive:
- **Append-only.** A client can add snapshots and nothing else: no delete, no
overwrite. The archive host removes old snapshots.
## External S3
By default the repositories sit on the archive host's disk. With
`deploy.archive.backend = "s3"` they sit in a bucket of an S3 service you run
or rent instead, and the host keeps nothing. Hives don't change: they still
talk to `archive.<swarm domain>` with their per-agent logins, and the archive
is still append-only. Only the archive holds the S3 key, so a compromised hive
can't reach the bucket directly.
| option | default | meaning |
| ---------------------------- | ------- | ----------------------------------------------------------------- |
| `deploy.archive.backend` | `local` | `local` or `s3` |
| `deploy.archive.s3.endpoint` | `null` | the service's URL, scheme included; required |
| `deploy.archive.s3.bucket` | `null` | the bucket; required, and it must exist |
| `deploy.archive.s3.prefix` | `""` | a path in the bucket; the repositories sit at `<prefix>/<agent>/` |
| `deploy.archive.s3.region` | `null` | the bucket's region, for services that need one |
| `deploy.archive.s3.provider` | `Other` | rclone's `--s3-provider`, for example `AWS` |
Evaluation fails if `backend` is `s3` and the endpoint or the bucket is unset.
The service needs one bucket and one access key that can list, read, write
and delete objects in it. Scope the key to the bucket. The archive never
creates a bucket, so the key needs no permission to. The prune removes old
data, which takes delete rights. Hives can't delete: the archive refuses it
before the request reaches the bucket.
The key lives in the secret store, like every other archive secret. Write it
before you set `backend = "s3"`; the server doesn't start without it. Read it
from a file or the terminal so it doesn't land in your shell history:
```sh
bao kv put secret/swarm/services/archive/s3/access-key-id value=-
bao kv put secret/swarm/services/archive/s3/secret-access-key value=-
```
Each command reads the value from standard input. The store's UI writes the
same two leaves, field `value`. The archive's host reads them as it starts and
passes them to rclone in its environment. They never reach a file there, so
a changed key takes effect at the next restart of `swarm-archive`. The daily
prune reads them on every run. If the server can't read them, it fails to
start and systemd retries every five seconds until the store answers.
Changing `backend` doesn't move the data. Before you switch an archive that
holds backups, copy them with `rclone copy <dataDir>/repos :s3:<bucket>/<prefix>`
using a rclone configured for the bucket, or start with empty repositories and
let the next hourly runs fill them. The old directory stays on the host until
you remove it.
## How long it stays
The archive host prunes every repository once a day. A snapshot survives if
@ -128,10 +177,14 @@ hives with the unit log that they have no credentials and succeed until it
updates. The archive host admits only the users the controller wrote, so until
then it admits nobody.
Where the repositories live is a setting of the archive host alone. Hives and `swarm-controller`
behave the same with either.
## What lives where
Everything secret lives in the swarm secret store, and the archive keeps none
on disk. The archive host renders its users file (password hashes only) into
`/run/swarm-archive-auth/`, which is tmpfs. Hives read their restic password
when restic asks for it. The paths and who writes and reads them are in the
`/run/swarm-archive-auth/`, which is tmpfs. With an external S3, it reads the
S3 key from the store when a process starts and passes it to rclone in the
environment. Hives read their restic password when restic asks for it. The paths and who writes and reads them are in the
[credentials reference](credentials-reference.md).