swarm-archive: external s3 backend (options, prune path, docs, module-eval cases, vm fake s3)
- swarm-archive.nix, lib/archive-server.nix: deploy.archive.backend and deploy.archive.s3.* options; the s3 server and the prune read the key from the store into rclone's environment; the prune lists and addresses repositories through rclone instead of a local directory - swarm-bao.nix: the archive reads swarm/services/archive/s3/* - docs: archive.md "External S3" section, two credentials-reference rows - module-eval: builder.nix cases for local and s3, the missing-setting assertion; bao-grants.nix gains the s3 read grant - vm test: second node with a fake rclone serve s3 whose key is generated at test time
This commit is contained in:
atlas
mara
parent
ebbf4fa37b
commit
86c5dac89d
2 changed files with 605 additions and 156 deletions
|
|
@ -33,8 +33,9 @@ as for the binary cache.
|
|||
|
||||
Each agent has its own [restic](https://restic.net) repository at
|
||||
`https://archive.<swarm domain>/<agent>/`, under `deploy.archive.dataDir`
|
||||
(default `/var/lib/swarm-archive`) on that host. restic encrypts it with a
|
||||
password of its own for that agent.
|
||||
(default `/var/lib/swarm-archive`) on that host, or in a bucket of an
|
||||
[external S3](#external-s3). restic encrypts it with a password of its own for
|
||||
that agent.
|
||||
|
||||
Two properties protect the backups from a compromised hive:
|
||||
|
||||
|
|
@ -49,6 +50,54 @@ Two properties protect the backups from a compromised hive:
|
|||
- **Append-only.** A client can add snapshots and nothing else: no delete, no
|
||||
overwrite. The archive host removes old snapshots.
|
||||
|
||||
## External S3
|
||||
|
||||
By default the repositories sit on the archive host's disk. With
|
||||
`deploy.archive.backend = "s3"` they sit in a bucket of an S3 service you run
|
||||
or rent instead, and the host keeps nothing. Hives don't change: they still
|
||||
talk to `archive.<swarm domain>` with their per-agent logins, and the archive
|
||||
is still append-only. Only the archive holds the S3 key, so a compromised hive
|
||||
can't reach the bucket directly.
|
||||
|
||||
| option | default | meaning |
|
||||
| ---------------------------- | ------- | ----------------------------------------------------------------- |
|
||||
| `deploy.archive.backend` | `local` | `local` or `s3` |
|
||||
| `deploy.archive.s3.endpoint` | `null` | the service's URL, scheme included; required |
|
||||
| `deploy.archive.s3.bucket` | `null` | the bucket; required, and it must exist |
|
||||
| `deploy.archive.s3.prefix` | `""` | a path in the bucket; the repositories sit at `<prefix>/<agent>/` |
|
||||
| `deploy.archive.s3.region` | `null` | the bucket's region, for services that need one |
|
||||
| `deploy.archive.s3.provider` | `Other` | rclone's `--s3-provider`, for example `AWS` |
|
||||
|
||||
Evaluation fails if `backend` is `s3` and the endpoint or the bucket is unset.
|
||||
|
||||
The service needs one bucket and one access key that can list, read, write
|
||||
and delete objects in it. Scope the key to the bucket. The archive never
|
||||
creates a bucket, so the key needs no permission to. The prune removes old
|
||||
data, which takes delete rights. Hives can't delete: the archive refuses it
|
||||
before the request reaches the bucket.
|
||||
|
||||
The key lives in the secret store, like every other archive secret. Write it
|
||||
before you set `backend = "s3"`; the server doesn't start without it. Read it
|
||||
from a file or the terminal so it doesn't land in your shell history:
|
||||
|
||||
```sh
|
||||
bao kv put secret/swarm/services/archive/s3/access-key-id value=-
|
||||
bao kv put secret/swarm/services/archive/s3/secret-access-key value=-
|
||||
```
|
||||
|
||||
Each command reads the value from standard input. The store's UI writes the
|
||||
same two leaves, field `value`. The archive's host reads them as it starts and
|
||||
passes them to rclone in its environment. They never reach a file there, so
|
||||
a changed key takes effect at the next restart of `swarm-archive`. The daily
|
||||
prune reads them on every run. If the server can't read them, it fails to
|
||||
start and systemd retries every five seconds until the store answers.
|
||||
|
||||
Changing `backend` doesn't move the data. Before you switch an archive that
|
||||
holds backups, copy them with `rclone copy <dataDir>/repos :s3:<bucket>/<prefix>`
|
||||
using a rclone configured for the bucket, or start with empty repositories and
|
||||
let the next hourly runs fill them. The old directory stays on the host until
|
||||
you remove it.
|
||||
|
||||
## How long it stays
|
||||
|
||||
The archive host prunes every repository once a day. A snapshot survives if
|
||||
|
|
@ -128,10 +177,14 @@ hives with the unit log that they have no credentials and succeed until it
|
|||
updates. The archive host admits only the users the controller wrote, so until
|
||||
then it admits nobody.
|
||||
|
||||
Where the repositories live is a setting of the archive host alone. Hives and `swarm-controller`
|
||||
behave the same with either.
|
||||
|
||||
## What lives where
|
||||
|
||||
Everything secret lives in the swarm secret store, and the archive keeps none
|
||||
on disk. The archive host renders its users file (password hashes only) into
|
||||
`/run/swarm-archive-auth/`, which is tmpfs. Hives read their restic password
|
||||
when restic asks for it. The paths and who writes and reads them are in the
|
||||
`/run/swarm-archive-auth/`, which is tmpfs. With an external S3, it reads the
|
||||
S3 key from the store when a process starts and passes it to rclone in the
|
||||
environment. Hives read their restic password when restic asks for it. The paths and who writes and reads them are in the
|
||||
[credentials reference](credentials-reference.md).
|
||||
|
|
|
|||
|
|
@ -61,6 +61,8 @@ the rest of the cell says how.
|
|||
| `swarm/services/archive/users/<agent>` (field `htpasswd`) | `swarm-controller`, with the REST password above: its `{SHA}` hash, never the password; deleted when no hive places the agent | `swarm-archive-users` on the archive's host, which renders the users file into `/run` | ✅ rewritten with the password | ✅ re-read every 2 minutes; the server notices the new file without a restart |
|
||||
| `swarm/services/archive/restic/<agent>` (field `password`) | `swarm-controller`, once per agent, on its first placement; never replaced and never deleted, because it opens every backup of the agent | `swarm-archive-prune` on the archive's host, through `RESTIC_PASSWORD_COMMAND` | ❌ a new password would not open the existing repository | ✅ read at every prune |
|
||||
| `swarm/services/archive/ready` | `swarm-archive-users` on the archive's host, once, when none is stored; `swarm-controller` writes no archive credentials before it exists | `swarm-controller`, every 5 minutes | ❌ never rewritten | ✅ read every 5 minutes |
|
||||
| `swarm/services/archive/s3/access-key-id` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive` and `swarm-archive-prune` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune |
|
||||
| `swarm/services/archive/s3/secret-access-key` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive` and `swarm-archive-prune` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune |
|
||||
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
|
||||
|
||||
|
|
@ -185,7 +187,7 @@ login minter's, the forge host's and Grafana's, which reach more than one:
|
|||
| `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services/<router client id>/oidc/client` |
|
||||
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services/<builder client id>/oidc/client` |
|
||||
| `swarm-bao-attic-secret` and `swarm-bao-attic-tokens` | `atticClientCertFile` / `atticClientKeyFile` | creates and reads `swarm/services/attic/jwt-secret`, `swarm/services/attic/push-token` and `swarm/services/attic/pull-token` |
|
||||
| `swarm-archive-users` and `swarm-archive-prune` | `archiveClientCertFile` / `archiveClientKeyFile` | reads `swarm/services/archive/users/*` and `swarm/services/archive/restic/*`, lists `swarm/services/archive/users/` |
|
||||
| `swarm-archive-users`, `swarm-archive-prune` and, with an S3 bucket, `swarm-archive` | `archiveClientCertFile` / `archiveClientKeyFile` | reads `swarm/services/archive/users/*`, `swarm/services/archive/restic/*` and `swarm/services/archive/s3/*`, lists `swarm/services/archive/users/` |
|
||||
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
|
||||
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue