swarm-archive: external s3 backend (options, prune path, docs, module-eval cases, vm fake s3)
- swarm-archive.nix, lib/archive-server.nix: deploy.archive.backend and deploy.archive.s3.* options; the s3 server and the prune read the key from the store into rclone's environment; the prune lists and addresses repositories through rclone instead of a local directory - swarm-bao.nix: the archive reads swarm/services/archive/s3/* - docs: archive.md "External S3" section, two credentials-reference rows - module-eval: builder.nix cases for local and s3, the missing-setting assertion; bao-grants.nix gains the s3 read grant - vm test: second node with a fake rclone serve s3 whose key is generated at test time
This commit is contained in:
atlas
mara
parent
ebbf4fa37b
commit
86c5dac89d
8 changed files with 605 additions and 156 deletions
|
|
@ -33,8 +33,9 @@ as for the binary cache.
|
|||
|
||||
Each agent has its own [restic](https://restic.net) repository at
|
||||
`https://archive.<swarm domain>/<agent>/`, under `deploy.archive.dataDir`
|
||||
(default `/var/lib/swarm-archive`) on that host. restic encrypts it with a
|
||||
password of its own for that agent.
|
||||
(default `/var/lib/swarm-archive`) on that host, or in a bucket of an
|
||||
[external S3](#external-s3). restic encrypts it with a password of its own for
|
||||
that agent.
|
||||
|
||||
Two properties protect the backups from a compromised hive:
|
||||
|
||||
|
|
@ -49,6 +50,54 @@ Two properties protect the backups from a compromised hive:
|
|||
- **Append-only.** A client can add snapshots and nothing else: no delete, no
|
||||
overwrite. The archive host removes old snapshots.
|
||||
|
||||
## External S3
|
||||
|
||||
By default the repositories sit on the archive host's disk. With
|
||||
`deploy.archive.backend = "s3"` they sit in a bucket of an S3 service you run
|
||||
or rent instead, and the host keeps nothing. Hives don't change: they still
|
||||
talk to `archive.<swarm domain>` with their per-agent logins, and the archive
|
||||
is still append-only. Only the archive holds the S3 key, so a compromised hive
|
||||
can't reach the bucket directly.
|
||||
|
||||
| option | default | meaning |
|
||||
| ---------------------------- | ------- | ----------------------------------------------------------------- |
|
||||
| `deploy.archive.backend` | `local` | `local` or `s3` |
|
||||
| `deploy.archive.s3.endpoint` | `null` | the service's URL, scheme included; required |
|
||||
| `deploy.archive.s3.bucket` | `null` | the bucket; required, and it must exist |
|
||||
| `deploy.archive.s3.prefix` | `""` | a path in the bucket; the repositories sit at `<prefix>/<agent>/` |
|
||||
| `deploy.archive.s3.region` | `null` | the bucket's region, for services that need one |
|
||||
| `deploy.archive.s3.provider` | `Other` | rclone's `--s3-provider`, for example `AWS` |
|
||||
|
||||
Evaluation fails if `backend` is `s3` and the endpoint or the bucket is unset.
|
||||
|
||||
The service needs one bucket and one access key that can list, read, write
|
||||
and delete objects in it. Scope the key to the bucket. The archive never
|
||||
creates a bucket, so the key needs no permission to. The prune removes old
|
||||
data, which takes delete rights. Hives can't delete: the archive refuses it
|
||||
before the request reaches the bucket.
|
||||
|
||||
The key lives in the secret store, like every other archive secret. Write it
|
||||
before you set `backend = "s3"`; the server doesn't start without it. Read it
|
||||
from a file or the terminal so it doesn't land in your shell history:
|
||||
|
||||
```sh
|
||||
bao kv put secret/swarm/services/archive/s3/access-key-id value=-
|
||||
bao kv put secret/swarm/services/archive/s3/secret-access-key value=-
|
||||
```
|
||||
|
||||
Each command reads the value from standard input. The store's UI writes the
|
||||
same two leaves, field `value`. The archive's host reads them as it starts and
|
||||
passes them to rclone in its environment. They never reach a file there, so
|
||||
a changed key takes effect at the next restart of `swarm-archive`. The daily
|
||||
prune reads them on every run. If the server can't read them, it fails to
|
||||
start and systemd retries every five seconds until the store answers.
|
||||
|
||||
Changing `backend` doesn't move the data. Before you switch an archive that
|
||||
holds backups, copy them with `rclone copy <dataDir>/repos :s3:<bucket>/<prefix>`
|
||||
using a rclone configured for the bucket, or start with empty repositories and
|
||||
let the next hourly runs fill them. The old directory stays on the host until
|
||||
you remove it.
|
||||
|
||||
## How long it stays
|
||||
|
||||
The archive host prunes every repository once a day. A snapshot survives if
|
||||
|
|
@ -128,10 +177,14 @@ hives with the unit log that they have no credentials and succeed until it
|
|||
updates. The archive host admits only the users the controller wrote, so until
|
||||
then it admits nobody.
|
||||
|
||||
Where the repositories live is a setting of the archive host alone. Hives and `swarm-controller`
|
||||
behave the same with either.
|
||||
|
||||
## What lives where
|
||||
|
||||
Everything secret lives in the swarm secret store, and the archive keeps none
|
||||
on disk. The archive host renders its users file (password hashes only) into
|
||||
`/run/swarm-archive-auth/`, which is tmpfs. Hives read their restic password
|
||||
when restic asks for it. The paths and who writes and reads them are in the
|
||||
`/run/swarm-archive-auth/`, which is tmpfs. With an external S3, it reads the
|
||||
S3 key from the store when a process starts and passes it to rclone in the
|
||||
environment. Hives read their restic password when restic asks for it. The paths and who writes and reads them are in the
|
||||
[credentials reference](credentials-reference.md).
|
||||
|
|
|
|||
|
|
@ -61,6 +61,8 @@ the rest of the cell says how.
|
|||
| `swarm/services/archive/users/<agent>` (field `htpasswd`) | `swarm-controller`, with the REST password above: its `{SHA}` hash, never the password; deleted when no hive places the agent | `swarm-archive-users` on the archive's host, which renders the users file into `/run` | ✅ rewritten with the password | ✅ re-read every 2 minutes; the server notices the new file without a restart |
|
||||
| `swarm/services/archive/restic/<agent>` (field `password`) | `swarm-controller`, once per agent, on its first placement; never replaced and never deleted, because it opens every backup of the agent | `swarm-archive-prune` on the archive's host, through `RESTIC_PASSWORD_COMMAND` | ❌ a new password would not open the existing repository | ✅ read at every prune |
|
||||
| `swarm/services/archive/ready` | `swarm-archive-users` on the archive's host, once, when none is stored; `swarm-controller` writes no archive credentials before it exists | `swarm-controller`, every 5 minutes | ❌ never rewritten | ✅ read every 5 minutes |
|
||||
| `swarm/services/archive/s3/access-key-id` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive` and `swarm-archive-prune` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune |
|
||||
| `swarm/services/archive/s3/secret-access-key` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive` and `swarm-archive-prune` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune |
|
||||
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
|
||||
|
||||
|
|
@ -185,7 +187,7 @@ login minter's, the forge host's and Grafana's, which reach more than one:
|
|||
| `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services/<router client id>/oidc/client` |
|
||||
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services/<builder client id>/oidc/client` |
|
||||
| `swarm-bao-attic-secret` and `swarm-bao-attic-tokens` | `atticClientCertFile` / `atticClientKeyFile` | creates and reads `swarm/services/attic/jwt-secret`, `swarm/services/attic/push-token` and `swarm/services/attic/pull-token` |
|
||||
| `swarm-archive-users` and `swarm-archive-prune` | `archiveClientCertFile` / `archiveClientKeyFile` | reads `swarm/services/archive/users/*` and `swarm/services/archive/restic/*`, lists `swarm/services/archive/users/` |
|
||||
| `swarm-archive-users`, `swarm-archive-prune` and, with an S3 bucket, `swarm-archive` | `archiveClientCertFile` / `archiveClientKeyFile` | reads `swarm/services/archive/users/*`, `swarm/services/archive/restic/*` and `swarm/services/archive/s3/*`, lists `swarm/services/archive/users/` |
|
||||
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
|
||||
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
|
||||
|
||||
|
|
|
|||
|
|
@ -5,16 +5,24 @@
|
|||
# (nix/vm-tests/swarm-archive.nix) runs the same scripts against a stand-in
|
||||
# `bao`.
|
||||
#
|
||||
# Both scripts use only `bao login -method=cert -token-only`,
|
||||
# All scripts use only `bao login -method=cert -token-only`,
|
||||
# `bao kv list -format=json` and `bao kv get -field=…`, and need `bao`, `jq`,
|
||||
# `coreutils`, `gnugrep` on the unit's `path` and the `BAO_*` client set.
|
||||
#
|
||||
# The repositories live in a directory (`reposDir`) or in an S3 bucket (`s3`).
|
||||
# Only the pieces that name the backend differ: the remote rclone serves, how
|
||||
# the prune lists and addresses the repositories, and the S3 key, which is read
|
||||
# from the secret store when a process starts.
|
||||
{ lib }:
|
||||
{
|
||||
rclone,
|
||||
restic,
|
||||
# Where the repositories live: `<reposDir>/<agent>/`. The only place that
|
||||
# names the backend; `rclone serve restic` takes any remote here.
|
||||
reposDir,
|
||||
# Local backend: the repositories live at `<reposDir>/<agent>/`. Null with `s3`.
|
||||
reposDir ? null,
|
||||
# S3 backend: `{ bucket, prefix, endpoint, region, provider }`, the
|
||||
# repositories at `<bucket>/<prefix>/<agent>/`; `region` may be null and
|
||||
# `prefix` empty. Null with `reposDir`.
|
||||
s3 ? null,
|
||||
# The address rclone listens on.
|
||||
addr,
|
||||
# The users file rclone checks logins against, its group and the user the
|
||||
|
|
@ -32,6 +40,8 @@ let
|
|||
usersPrefix = "secret/swarm/services/archive/users";
|
||||
resticPrefix = "secret/swarm/services/archive/restic";
|
||||
readyPath = "secret/swarm/services/archive/ready";
|
||||
s3KeyIdPath = "secret/swarm/services/archive/s3/access-key-id";
|
||||
s3SecretKeyPath = "secret/swarm/services/archive/s3/secret-access-key";
|
||||
|
||||
# Slack for the clocks of the hives that stamp snapshots.
|
||||
futureSkewSeconds = 3600;
|
||||
|
|
@ -42,13 +52,48 @@ let
|
|||
retention.keepWithin != null
|
||||
) "--keep-within ${lib.escapeShellArg retention.keepWithin}"
|
||||
);
|
||||
in
|
||||
{
|
||||
inherit usersPrefix resticPrefix;
|
||||
|
||||
# Append-only: a client cannot delete or overwrite a snapshot, a pack or the
|
||||
# index, so a hive that is compromised cannot erase history. Private repos:
|
||||
# a login reaches `/<its own name>/` and nothing else.
|
||||
# An rclone connection string, so no rclone config exists; the connection
|
||||
# settings are `environment` below.
|
||||
s3Remote = ":s3:${s3.bucket}${lib.optionalString (s3.prefix != "") "/${s3.prefix}"}";
|
||||
|
||||
# What rclone is told to serve.
|
||||
backendArg = if s3 == null then reposDir else s3Remote;
|
||||
|
||||
# Reads the S3 key from the store into rclone's environment. Needs the
|
||||
# caller's store login.
|
||||
s3Credentials = ''
|
||||
RCLONE_S3_ACCESS_KEY_ID="$(timeout 15 bao kv get -field=value ${lib.escapeShellArg s3KeyIdPath})"
|
||||
RCLONE_S3_SECRET_ACCESS_KEY="$(timeout 15 bao kv get -field=value ${lib.escapeShellArg s3SecretKeyPath})"
|
||||
export RCLONE_S3_ACCESS_KEY_ID RCLONE_S3_SECRET_ACCESS_KEY
|
||||
'';
|
||||
|
||||
# Prints the name of every agent that has a repository, one per line.
|
||||
listAgents =
|
||||
if s3 == null then
|
||||
''
|
||||
for repo in ${lib.escapeShellArg reposDir}/*/; do
|
||||
[ -e "$repo/config" ] || continue
|
||||
basename -- "$repo"
|
||||
done
|
||||
''
|
||||
else
|
||||
''
|
||||
rclone lsf --dirs-only ${lib.escapeShellArg "${s3Remote}/"} | while read -r dir; do
|
||||
dir="''${dir%/}"
|
||||
[ -n "$(rclone lsf ${lib.escapeShellArg s3Remote}/"$dir"/config)" ] || continue
|
||||
printf '%s\n' "$dir"
|
||||
done
|
||||
'';
|
||||
|
||||
# restic's name for the repository of `$agent`: the directory, or the rclone
|
||||
# backend over the same remote the server serves.
|
||||
repoOf =
|
||||
if s3 == null then
|
||||
''${lib.escapeShellArg reposDir}/"$agent"/''
|
||||
else
|
||||
''${lib.escapeShellArg "rclone:${s3Remote}/"}"$agent"'';
|
||||
|
||||
serverCommand = lib.escapeShellArgs [
|
||||
"${rclone}/bin/rclone"
|
||||
"serve"
|
||||
|
|
@ -59,8 +104,41 @@ in
|
|||
"--private-repos"
|
||||
"--htpasswd"
|
||||
authFile
|
||||
reposDir
|
||||
backendArg
|
||||
];
|
||||
in
|
||||
{
|
||||
inherit usersPrefix resticPrefix;
|
||||
|
||||
# The S3 remote's connection settings, everything but the key. Empty for the
|
||||
# local backend.
|
||||
environment = lib.optionalAttrs (s3 != null) (
|
||||
{
|
||||
RCLONE_S3_PROVIDER = s3.provider;
|
||||
RCLONE_S3_ENDPOINT = s3.endpoint;
|
||||
# A key scoped to the bucket cannot create buckets.
|
||||
RCLONE_S3_NO_CHECK_BUCKET = "true";
|
||||
RCLONE_S3_ENV_AUTH = "false";
|
||||
}
|
||||
// lib.optionalAttrs (s3.region != null) { RCLONE_S3_REGION = s3.region; }
|
||||
);
|
||||
|
||||
# Append-only: a client cannot delete or overwrite a snapshot, a pack or the
|
||||
# index, so a hive that is compromised cannot erase history. Private repos:
|
||||
# a login reaches `/<its own name>/` and nothing else.
|
||||
inherit serverCommand;
|
||||
|
||||
# Start of the S3 server: logs in with the unit's store identity, puts the key
|
||||
# in rclone's environment and becomes rclone. The key is never written out. A
|
||||
# store that cannot answer fails the start, and the unit's restart retries.
|
||||
s3ServerScript = ''
|
||||
set -euo pipefail
|
||||
BAO_TOKEN="$(timeout 15 bao login -method=cert -token-only)"
|
||||
export BAO_TOKEN
|
||||
${s3Credentials}
|
||||
unset BAO_TOKEN
|
||||
exec ${serverCommand}
|
||||
'';
|
||||
|
||||
# Renders the users file from `usersPrefix`, one leaf per agent holding that
|
||||
# agent's htpasswd line. The file is written only when the content changed,
|
||||
|
|
@ -171,15 +249,21 @@ in
|
|||
exit 1
|
||||
fi
|
||||
|
||||
${lib.optionalString (s3 != null) s3Credentials}
|
||||
list_agents() {
|
||||
${listAgents}
|
||||
}
|
||||
# Under `set -e` a failed listing ends the run; inside the `for` it would not.
|
||||
agents="$(list_agents)"
|
||||
|
||||
failed=0
|
||||
for repo in ${lib.escapeShellArg reposDir}/*/; do
|
||||
[ -e "$repo/config" ] || continue
|
||||
agent="$(basename -- "$repo")"
|
||||
for agent in $agents; do
|
||||
if [[ ! "$agent" =~ ^[A-Za-z0-9_-]+$ ]]; then
|
||||
echo "archive prune failed: $agent is not an agent name" >&2
|
||||
failed=1
|
||||
continue
|
||||
fi
|
||||
repo=${repoOf}
|
||||
export RESTIC_PASSWORD_COMMAND="bao kv get -field=password ${resticPrefix}/$agent"
|
||||
if forget_future "$repo" "$agent" \
|
||||
&& ${restic}/bin/restic -r "$repo" --retry-lock 10m forget --group-by host ${keepFlags} --prune; then
|
||||
|
|
|
|||
|
|
@ -23,11 +23,11 @@
|
|||
# therefore decrypt every backup, which is the cost of pruning where clients
|
||||
# cannot delete.
|
||||
#
|
||||
# Both units log in as the archive's own store principal (a leaf of the store's
|
||||
# The units log in as the archive's own store principal (a leaf of the store's
|
||||
# CA, minted by ./glue-bao-tls.nix). Plain host units rather than a container:
|
||||
# no secret crosses a mount, and the backend is one string
|
||||
# (`reposDir`, in ./lib/archive-server.nix) that names where the repositories
|
||||
# are.
|
||||
# no secret crosses a mount. The backend is one argument (`reposDir` or `s3`, in
|
||||
# ./lib/archive-server.nix); with `s3` the server also reads the S3 key from
|
||||
# `swarm/services/archive/s3/…` as it starts and keeps it in rclone's environment.
|
||||
{
|
||||
pkgs,
|
||||
lib,
|
||||
|
|
@ -51,17 +51,40 @@ let
|
|||
authFile = "${authDir}/htpasswd";
|
||||
reposDir = "${cfg.dataDir}/repos";
|
||||
|
||||
server = import ./lib/archive-server.nix { inherit lib; } {
|
||||
inherit (cfg) retention;
|
||||
inherit authFile reposDir;
|
||||
rclone = cfg.package;
|
||||
restic = cfg.resticPackage;
|
||||
addr = "127.0.0.1:${toString cfg.port}";
|
||||
authGroup = user;
|
||||
};
|
||||
useS3 = cfg.backend == "s3";
|
||||
|
||||
server = import ./lib/archive-server.nix { inherit lib; } (
|
||||
{
|
||||
inherit (cfg) retention;
|
||||
inherit authFile;
|
||||
rclone = cfg.package;
|
||||
restic = cfg.resticPackage;
|
||||
addr = "127.0.0.1:${toString cfg.port}";
|
||||
authGroup = user;
|
||||
}
|
||||
// (if useS3 then { inherit (cfg) s3; } else { inherit reposDir; })
|
||||
);
|
||||
|
||||
storeRetry = import ./lib/store-retry.nix { };
|
||||
|
||||
# A unit that runs as the repositories' owner gets the store identity as
|
||||
# credentials, which only that user can read.
|
||||
ownerStoreCredentials = [
|
||||
"bao-cert:${baoDeploy.archiveClientCertFile}"
|
||||
"bao-key:${baoDeploy.archiveClientKeyFile}"
|
||||
]
|
||||
++ lib.optional (baoDeploy.serverCaFile != null) "bao-ca:${baoDeploy.serverCaFile}";
|
||||
ownerStoreEnvironment = {
|
||||
BAO_ADDR = storeEnvironment.BAO_ADDR;
|
||||
BAO_CLIENT_CERT = "%d/bao-cert";
|
||||
BAO_CLIENT_KEY = "%d/bao-key";
|
||||
}
|
||||
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
|
||||
BAO_CACERT = "%d/bao-ca";
|
||||
};
|
||||
|
||||
serveScript = pkgs.writeShellScript "swarm-archive-serve" server.s3ServerScript;
|
||||
|
||||
haveClientIdentity =
|
||||
baoDeploy.archiveClientCertFile != null && baoDeploy.archiveClientKeyFile != null;
|
||||
|
||||
|
|
@ -101,6 +124,72 @@ in
|
|||
description = "The restic package the prune unit runs.";
|
||||
};
|
||||
|
||||
backend = lib.mkOption {
|
||||
type = lib.types.enum [
|
||||
"local"
|
||||
"s3"
|
||||
];
|
||||
default = "local";
|
||||
description = ''
|
||||
Where the repositories are kept: `local` is a directory on this host
|
||||
({option}`services.hyperhive.deploy.archive.dataDir`), `s3` a bucket of an
|
||||
external S3 service (`services.hyperhive.deploy.archive.s3.*`).
|
||||
Hives always talk to this archive, whatever the backend, and never see the
|
||||
S3 key. The archive reads the key from the secret store, in the leaves
|
||||
`swarm/services/archive/s3/access-key-id` and `…/secret-access-key`
|
||||
(field `value`), which you write.
|
||||
|
||||
Switching does not move the repositories already stored. Copy them first
|
||||
(`rclone copy <dataDir>/repos :s3:<bucket>/<prefix>`) or start with
|
||||
empty repositories.
|
||||
'';
|
||||
};
|
||||
|
||||
s3 = {
|
||||
endpoint = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "https://s3.eu-central-1.amazonaws.com";
|
||||
description = "URL of the S3 service, scheme included. Required with `backend = \"s3\"`.";
|
||||
};
|
||||
|
||||
bucket = lib.mkOption {
|
||||
type = lib.types.nullOr (lib.types.strMatching "[A-Za-z0-9._-]+");
|
||||
default = null;
|
||||
description = ''
|
||||
The bucket that holds the repositories. Required with `backend = "s3"`.
|
||||
It must exist already: the archive creates none.
|
||||
'';
|
||||
};
|
||||
|
||||
prefix = lib.mkOption {
|
||||
type = lib.types.strMatching "([A-Za-z0-9._-]+(/[A-Za-z0-9._-]+)*)?";
|
||||
default = "";
|
||||
example = "archive";
|
||||
description = ''
|
||||
A path in the bucket, without leading or trailing slash. The
|
||||
repositories sit at `<prefix>/<agent>/`; empty puts them at the top.
|
||||
'';
|
||||
};
|
||||
|
||||
region = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "eu-central-1";
|
||||
description = "The bucket's region, for services that require one. `null` leaves rclone's default.";
|
||||
};
|
||||
|
||||
provider = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "Other";
|
||||
example = "AWS";
|
||||
description = ''
|
||||
rclone's `--s3-provider` value, which selects the service's quirks.
|
||||
`Other` suits a generic S3-compatible service.
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
port = lib.mkOption {
|
||||
type = lib.types.port;
|
||||
default = 8911;
|
||||
|
|
@ -114,8 +203,8 @@ in
|
|||
type = lib.types.path;
|
||||
default = "/var/lib/swarm-archive";
|
||||
description = ''
|
||||
Where the repositories live, under `repos/<agent>/`. Holds restic
|
||||
ciphertext: every backup of every agent, so size it for that.
|
||||
Where the repositories live with `backend = "local"`, under `repos/<agent>/`.
|
||||
Holds restic ciphertext: every backup of every agent, so size it for that.
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
@ -172,6 +261,14 @@ in
|
|||
host's /var/lib/swarm-bao-pki/ and name them in those two options.
|
||||
'';
|
||||
}
|
||||
{
|
||||
assertion = !useS3 || (cfg.s3.endpoint != null && cfg.s3.bucket != null);
|
||||
message = ''
|
||||
services.hyperhive.deploy.archive.backend = "s3" requires both
|
||||
services.hyperhive.deploy.archive.s3.endpoint and
|
||||
services.hyperhive.deploy.archive.s3.bucket.
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
||||
users.users.${user} = {
|
||||
|
|
@ -212,9 +309,11 @@ in
|
|||
];
|
||||
|
||||
systemd.tmpfiles.rules = [
|
||||
"d ${authDir} 0750 root ${user} -"
|
||||
]
|
||||
++ lib.optionals (!useS3) [
|
||||
"d ${cfg.dataDir} 0750 ${user} ${user} -"
|
||||
"d ${reposDir} 0750 ${user} ${user} -"
|
||||
"d ${authDir} 0750 root ${user} -"
|
||||
];
|
||||
|
||||
systemd.services.swarm-archive = {
|
||||
|
|
@ -227,9 +326,16 @@ in
|
|||
"systemd-tmpfiles-setup.service"
|
||||
];
|
||||
# rclone writes no config, and reads none.
|
||||
environment.RCLONE_CONFIG = "/dev/null";
|
||||
environment = {
|
||||
RCLONE_CONFIG = "/dev/null";
|
||||
}
|
||||
// lib.optionalAttrs useS3 (ownerStoreEnvironment // server.environment);
|
||||
path = lib.optionals useS3 [
|
||||
baoDeploy.package
|
||||
pkgs.coreutils
|
||||
];
|
||||
serviceConfig = {
|
||||
ExecStart = server.serverCommand;
|
||||
ExecStart = if useS3 then serveScript else server.serverCommand;
|
||||
User = user;
|
||||
Group = user;
|
||||
Restart = "on-failure";
|
||||
|
|
@ -246,9 +352,10 @@ in
|
|||
"AF_INET"
|
||||
"AF_INET6"
|
||||
];
|
||||
ReadWritePaths = [ reposDir ];
|
||||
ReadOnlyPaths = [ authDir ];
|
||||
};
|
||||
}
|
||||
// lib.optionalAttrs useS3 { LoadCredential = ownerStoreCredentials; }
|
||||
// lib.optionalAttrs (!useS3) { ReadWritePaths = [ reposDir ]; };
|
||||
};
|
||||
|
||||
# The users file, from the store. A failed run leaves the file as it is;
|
||||
|
|
@ -303,37 +410,30 @@ in
|
|||
pkgs.coreutils
|
||||
pkgs.gnugrep
|
||||
pkgs.jq
|
||||
];
|
||||
# The unit runs as the repositories' owner, so the store identity comes
|
||||
# in as credentials, which only it can read.
|
||||
environment = {
|
||||
BAO_ADDR = storeEnvironment.BAO_ADDR;
|
||||
BAO_CLIENT_CERT = "%d/bao-cert";
|
||||
BAO_CLIENT_KEY = "%d/bao-key";
|
||||
RESTIC_CACHE_DIR = "%C/swarm-archive-prune";
|
||||
}
|
||||
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
|
||||
BAO_CACERT = "%d/bao-ca";
|
||||
};
|
||||
]
|
||||
# restic's rclone backend starts rclone.
|
||||
++ lib.optional useS3 cfg.package;
|
||||
environment =
|
||||
ownerStoreEnvironment
|
||||
// {
|
||||
RESTIC_CACHE_DIR = "%C/swarm-archive-prune";
|
||||
}
|
||||
// lib.optionalAttrs useS3 ({ RCLONE_CONFIG = "/dev/null"; } // server.environment);
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
User = user;
|
||||
Group = user;
|
||||
SyslogIdentifier = "swarm-archive-prune";
|
||||
CacheDirectory = "swarm-archive-prune";
|
||||
LoadCredential = [
|
||||
"bao-cert:${baoDeploy.archiveClientCertFile}"
|
||||
"bao-key:${baoDeploy.archiveClientKeyFile}"
|
||||
]
|
||||
++ lib.optional (baoDeploy.serverCaFile != null) "bao-ca:${baoDeploy.serverCaFile}";
|
||||
LoadCredential = ownerStoreCredentials;
|
||||
NoNewPrivileges = true;
|
||||
ProtectSystem = "strict";
|
||||
ProtectHome = true;
|
||||
PrivateTmp = true;
|
||||
PrivateDevices = true;
|
||||
ReadWritePaths = [ reposDir ];
|
||||
TimeoutStartSec = "6h";
|
||||
};
|
||||
}
|
||||
// lib.optionalAttrs (!useS3) { ReadWritePaths = [ reposDir ]; };
|
||||
script = server.pruneScript;
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -1394,7 +1394,8 @@ let
|
|||
# The backup archive: the users file is rendered from one leaf per agent, and
|
||||
# the prune reads each agent's restic password. Read and list only; the
|
||||
# controller writes both. It also creates the marker that tells the
|
||||
# controller an archive exists.
|
||||
# controller an archive exists. With an external S3 backend it reads the
|
||||
# provider key, which the operator writes.
|
||||
archiveReaders = [
|
||||
{
|
||||
name = "swarm-archive";
|
||||
|
|
@ -1402,6 +1403,7 @@ let
|
|||
policyText =
|
||||
readStanza "${credentialMountPath}/data/swarm/services/archive/users/*"
|
||||
+ readStanza "${credentialMountPath}/data/swarm/services/archive/restic/*"
|
||||
+ readStanza "${credentialMountPath}/data/swarm/services/archive/s3/*"
|
||||
+ listStanza "${credentialMountPath}/metadata/swarm/services/archive/users/"
|
||||
+ createReadStanza "${credentialMountPath}/data/swarm/services/archive/ready";
|
||||
}
|
||||
|
|
@ -2685,7 +2687,7 @@ in
|
|||
description = ''
|
||||
Subject the store's `swarm-archive` cert-auth role accepts: the identity
|
||||
the backup archive's host units present. Its grant is read on
|
||||
`swarm/services/archive/users/*` and `restic/*`, and list on `users`.
|
||||
`swarm/services/archive/users/*`, `restic/*` and `s3/*`, and list on `users`.
|
||||
|
||||
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
|
||||
'';
|
||||
|
|
|
|||
|
|
@ -2201,6 +2201,9 @@ let
|
|||
in
|
||||
(grantFor grants "secret/data/swarm/services/archive/users/atlas").caps == [ "read" ]
|
||||
&& (grantFor grants "secret/data/swarm/services/archive/restic/atlas").caps == [ "read" ]
|
||||
&& (grantFor grants "secret/data/swarm/services/archive/s3/access-key-id").caps == [ "read" ]
|
||||
&& (grantFor grants "secret/data/swarm/services/archive/s3/secret-access-key").caps == [ "read" ]
|
||||
&& grantFor c "secret/data/swarm/services/archive/s3/access-key-id" == null
|
||||
&& (grantFor grants "secret/metadata/swarm/services/archive/users/").caps == [ "list" ]
|
||||
&&
|
||||
(grantFor grants "secret/data/swarm/services/archive/ready").caps == [
|
||||
|
|
@ -2209,7 +2212,7 @@ let
|
|||
]
|
||||
&& (grantFor c "secret/data/swarm/services/archive/ready").caps == [ "read" ]
|
||||
&& grantFor grants "secret/data/swarm/hives/h1/archive/atlas" == null
|
||||
&& lib.length grants == 4
|
||||
&& lib.length grants == 5
|
||||
&& lib.hasInfix "allowed_common_names=swarm-archive" s
|
||||
&& lib.hasInfix "token_policies=swarm-archive" s
|
||||
&& lib.elem "create" (grantFor c "secret/data/swarm/services/archive/users/atlas").caps
|
||||
|
|
|
|||
|
|
@ -46,6 +46,8 @@ let
|
|||
deploy.builder.enable = true;
|
||||
swarm.attic.domain = "cache.example.org";
|
||||
swarm.attic.cache = null;
|
||||
deploy.archive.enable = true;
|
||||
deploy.archive.backend = "s3";
|
||||
};
|
||||
refusedHalfCache =
|
||||
m:
|
||||
|
|
@ -109,6 +111,13 @@ let
|
|||
deploy.attic.enable = lib.mkForce false;
|
||||
swarm.attic.domain = "cache.example.org";
|
||||
swarm.attic.cache = "grid";
|
||||
deploy.archive.backend = "s3";
|
||||
deploy.archive.s3 = {
|
||||
endpoint = "https://s3.example.org";
|
||||
region = "eu-central-1";
|
||||
bucket = "bkt";
|
||||
prefix = "archive";
|
||||
};
|
||||
};
|
||||
atticWarnings =
|
||||
m:
|
||||
|
|
@ -600,6 +609,102 @@ let
|
|||
&& !(bare.systemd.services ? swarm-archive)
|
||||
&& !(bare.systemd.services ? swarm-archive-users);
|
||||
}
|
||||
{
|
||||
# The default is the directory backend, and its unit is the one that ran
|
||||
# before the S3 option: the literal command line is the control.
|
||||
name = "the archive's default backend is the local directory and renders the unchanged server unit";
|
||||
ok =
|
||||
let
|
||||
a = atticHere.systemd.services.swarm-archive;
|
||||
prune = atticHere.systemd.services.swarm-archive-prune;
|
||||
cfg = atticHere.services.hyperhive.deploy.archive;
|
||||
in
|
||||
cfg.backend == "local"
|
||||
&&
|
||||
a.serviceConfig.ExecStart
|
||||
== "${cfg.package}/bin/rclone serve restic --addr 127.0.0.1:8911 --append-only --private-repos --htpasswd /run/swarm-archive-auth/htpasswd /var/lib/swarm-archive/repos"
|
||||
&& a.environment.RCLONE_CONFIG == "/dev/null"
|
||||
&& !(lib.any (n: lib.hasPrefix "RCLONE_S3" n) (lib.attrNames a.environment))
|
||||
&& a.serviceConfig.ReadWritePaths == [ "/var/lib/swarm-archive/repos" ]
|
||||
&& !(a.serviceConfig ? LoadCredential)
|
||||
&& prune.serviceConfig.ReadWritePaths == [ "/var/lib/swarm-archive/repos" ]
|
||||
&& lib.hasInfix "/var/lib/swarm-archive/repos" prune.script
|
||||
&& !(lib.hasInfix "rclone" prune.script)
|
||||
&& lib.elem "d /var/lib/swarm-archive/repos 0750 swarm-archive swarm-archive -" atticHere.systemd.tmpfiles.rules;
|
||||
}
|
||||
{
|
||||
# The key is read as the server starts and lives in rclone's environment:
|
||||
# the unit carries the connection settings and the store identity, never
|
||||
# the key, and the prune reaches the bucket through restic's rclone
|
||||
# backend. The local case above is the control for each absence.
|
||||
name = "the archive on an external S3 serves and prunes the bucket, reads its key from the store at start and keeps it out of the unit";
|
||||
ok =
|
||||
let
|
||||
a = atticExternal.systemd.services.swarm-archive;
|
||||
prune = atticExternal.systemd.services.swarm-archive-prune;
|
||||
cfg = atticExternal.services.hyperhive.deploy.archive;
|
||||
server = import ../host-modules/lib/archive-server.nix { inherit lib; } {
|
||||
rclone = pkgs.rclone;
|
||||
restic = pkgs.restic;
|
||||
addr = "127.0.0.1:8911";
|
||||
authFile = "/run/swarm-archive-auth/htpasswd";
|
||||
authGroup = "swarm-archive";
|
||||
retention = {
|
||||
keepWithin = "3d";
|
||||
keepLast = 3;
|
||||
};
|
||||
s3 = cfg.s3;
|
||||
};
|
||||
unitText = builtins.toJSON [
|
||||
a.environment
|
||||
a.serviceConfig
|
||||
prune.environment
|
||||
prune.serviceConfig
|
||||
];
|
||||
in
|
||||
cfg.backend == "s3"
|
||||
&& lib.hasInfix "swarm-archive-serve" a.serviceConfig.ExecStart
|
||||
&& a.environment.RCLONE_S3_ENDPOINT == "https://s3.example.org"
|
||||
&& a.environment.RCLONE_S3_REGION == "eu-central-1"
|
||||
&& a.environment.RCLONE_S3_PROVIDER == "Other"
|
||||
&& a.environment.RCLONE_S3_NO_CHECK_BUCKET == "true"
|
||||
&& a.environment.RCLONE_S3_ENV_AUTH == "false"
|
||||
&& a.environment.RCLONE_CONFIG == "/dev/null"
|
||||
&& a.environment.BAO_CLIENT_CERT == "%d/bao-cert"
|
||||
&& lib.elem "bao-cert:/var/lib/swarm-bao-pki/archive.pem" a.serviceConfig.LoadCredential
|
||||
&& !(a.serviceConfig ? ReadWritePaths)
|
||||
&& !(prune.serviceConfig ? ReadWritePaths)
|
||||
&& prune.environment.RCLONE_S3_ENDPOINT == "https://s3.example.org"
|
||||
&& !(lib.elem "d /var/lib/swarm-archive/repos 0750 swarm-archive swarm-archive -" atticExternal.systemd.tmpfiles.rules)
|
||||
&& lib.hasInfix "RCLONE_S3_ENDPOINT" unitText
|
||||
&& !(lib.hasInfix "ACCESS_KEY" unitText)
|
||||
&& lib.hasInfix "--append-only" server.serverCommand
|
||||
&& lib.hasInfix "--private-repos" server.serverCommand
|
||||
&& lib.hasSuffix " :s3:bkt/archive" server.serverCommand
|
||||
&& lib.hasInfix "bao kv get -field=value secret/swarm/services/archive/s3/access-key-id" server.s3ServerScript
|
||||
&& lib.hasInfix "bao kv get -field=value secret/swarm/services/archive/s3/secret-access-key" server.s3ServerScript
|
||||
&& lib.hasInfix "\nexec /nix/store/" server.s3ServerScript
|
||||
&& lib.hasInfix "/bin/rclone serve restic" server.s3ServerScript
|
||||
&& lib.hasInfix "rclone lsf --dirs-only :s3:bkt/archive/" server.pruneScript
|
||||
&& lib.hasInfix "repo=rclone::s3:bkt/archive/\"$agent\"" server.pruneScript
|
||||
&& lib.hasInfix "RCLONE_S3_SECRET_ACCESS_KEY" server.pruneScript
|
||||
&& !(lib.hasInfix "/var/lib/swarm-archive" server.pruneScript);
|
||||
}
|
||||
{
|
||||
# `halfCache` hosts the archive on the S3 backend with neither endpoint nor
|
||||
# bucket; `atticExternal` names both and is the control that the
|
||||
# assertion fires for the missing settings alone.
|
||||
name = "the archive on S3 without an endpoint and a bucket fails evaluation, and with both it does not";
|
||||
ok =
|
||||
let
|
||||
refused =
|
||||
m:
|
||||
lib.any (
|
||||
a: !a.assertion && lib.hasInfix "deploy.archive.backend = \"s3\" requires" a.message
|
||||
) m.assertions;
|
||||
in
|
||||
refused halfCache && !(refused atticExternal);
|
||||
}
|
||||
];
|
||||
in
|
||||
runGroup "builder" cases
|
||||
|
|
|
|||
|
|
@ -1,7 +1,9 @@
|
|||
# `vmTests.<system>.swarm-archive`: runs the archive server's real units
|
||||
# (../host-modules/swarm-archive.nix) and the hive's real backup script
|
||||
# (../host-modules/lib/archive-backup.nix) in a VM, with restic and rclone, and
|
||||
# checks what the module-eval suite can only read:
|
||||
# checks what the module-eval suite can only read, on two nodes: `machine` keeps
|
||||
# the repositories in a directory, `s3archive` in a bucket of a fake S3
|
||||
# (`rclone serve s3`). On both:
|
||||
#
|
||||
# - a backup of two agents lands in one repository each, and `work/` is left out;
|
||||
# - one agent's login cannot read another's repository;
|
||||
|
|
@ -9,14 +11,18 @@
|
|||
# applies the retention, also when a client has forged a snapshot dated in
|
||||
# the future;
|
||||
# - the users file follows the store: a user removed from it is refused without
|
||||
# a restart of the server;
|
||||
# - a hive that has agents and no grant backs up nothing and succeeds.
|
||||
# a restart of the server.
|
||||
#
|
||||
# One node, because the gateway hop is outside what is under test: the
|
||||
# backup script dials `http://127.0.0.1:<port>` instead of
|
||||
# `https://archive.<domain>`, and the swarm secret store is a stand-in `bao`
|
||||
# that serves `kv list`, `kv get` and `login` from a directory. The module-eval
|
||||
# suite covers the gateway vhost and the store grants.
|
||||
# Only on `machine`: a hive that has agents and no grant backs up nothing and
|
||||
# succeeds. Only on `s3archive`: the server does not start while the store
|
||||
# holds no S3 key; the repositories are objects in the bucket and not on the
|
||||
# host; the key is in rclone's environment and in no unit or file.
|
||||
#
|
||||
# The gateway hop is outside what is under test: the backup script dials
|
||||
# `http://127.0.0.1:<port>` instead of `https://archive.<domain>`, and the swarm
|
||||
# secret store is a stand-in `bao` that serves `kv list`, `kv get` and `login`
|
||||
# from a directory. The module-eval suite covers the gateway vhost and the store
|
||||
# grants.
|
||||
{ pkgs }:
|
||||
let
|
||||
port = 8911;
|
||||
|
|
@ -135,16 +141,26 @@ let
|
|||
excludes = [ ];
|
||||
caBundle = null;
|
||||
};
|
||||
in
|
||||
pkgs.testers.runNixOSTest {
|
||||
name = "swarm-archive";
|
||||
|
||||
nodes.machine =
|
||||
# The external S3 of the second node: `rclone serve s3` over a directory,
|
||||
# whose sub-directories are the buckets. The archive reaches it over HTTP
|
||||
# with a key the unit generates at start, as it would reach a provider. rclone
|
||||
# refuses short keys.
|
||||
s3Port = 9000;
|
||||
s3Root = "/var/lib/fake-s3";
|
||||
s3Bucket = "archive-bucket";
|
||||
s3KeyDir = "/var/lib/fake-s3-key";
|
||||
|
||||
# What both nodes share: the archive module, the stubs for the hive options it
|
||||
# reads, and the hive's backup unit.
|
||||
archiveNode =
|
||||
extra:
|
||||
{ pkgs, ... }:
|
||||
{
|
||||
imports = [
|
||||
hiveStubs
|
||||
../host-modules/swarm-archive.nix
|
||||
extra
|
||||
];
|
||||
|
||||
# The newest snapshot, plus whatever is younger than three days.
|
||||
|
|
@ -174,23 +190,56 @@ pkgs.testers.runNixOSTest {
|
|||
script = backupScript;
|
||||
};
|
||||
};
|
||||
in
|
||||
pkgs.testers.runNixOSTest {
|
||||
name = "swarm-archive";
|
||||
|
||||
nodes.machine = archiveNode { };
|
||||
|
||||
nodes.s3archive = archiveNode {
|
||||
services.hyperhive.deploy.archive = {
|
||||
backend = "s3";
|
||||
s3 = {
|
||||
endpoint = "http://127.0.0.1:${toString s3Port}";
|
||||
bucket = s3Bucket;
|
||||
prefix = "pfx";
|
||||
region = "us-east-1";
|
||||
};
|
||||
};
|
||||
|
||||
systemd.services.fake-s3 = {
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
before = [ "swarm-archive.service" ];
|
||||
environment.RCLONE_CONFIG = "/dev/null";
|
||||
script = ''
|
||||
mkdir -p ${s3Root}/${s3Bucket}
|
||||
install -d -m 0700 ${s3KeyDir}
|
||||
if [ ! -e ${s3KeyDir}/id ]; then
|
||||
head -c 15 /dev/urandom | base32 | tr -d = > ${s3KeyDir}/id
|
||||
head -c 24 /dev/urandom | base32 | tr -d = > ${s3KeyDir}/secret
|
||||
fi
|
||||
exec ${pkgs.rclone}/bin/rclone serve s3 --addr 127.0.0.1:${toString s3Port} \
|
||||
--auth-key "$(cat ${s3KeyDir}/id),$(cat ${s3KeyDir}/secret)" ${s3Root}
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
testScript = ''
|
||||
store = "${store}"
|
||||
url = "http://127.0.0.1:${toString port}"
|
||||
agents = ["alpha", "beta"]
|
||||
|
||||
def sha_line(agent, password):
|
||||
return machine.succeed(
|
||||
def sha_line(m, agent, password):
|
||||
return m.succeed(
|
||||
f"printf '%s:{{SHA}}%s' {agent} "
|
||||
f"\"$(printf %s {password} | openssl dgst -sha1 -binary | base64)\""
|
||||
)
|
||||
|
||||
def put(path, field, value):
|
||||
machine.succeed(f"mkdir -p {store}/{path} && printf %s '{value}' > {store}/{path}/{field}")
|
||||
def put(m, path, field, value):
|
||||
m.succeed(f"mkdir -p {store}/{path} && printf %s '{value}' > {store}/{path}/{field}")
|
||||
|
||||
def status(agent, repo, path="config", method="GET"):
|
||||
return machine.succeed(
|
||||
def status(m, agent, repo, path="config", method="GET"):
|
||||
return m.succeed(
|
||||
f"curl -s -o /dev/null -w '%{{http_code}}' -X {method} "
|
||||
f"-u {agent}:restpw-{agent} {url}/{repo}/{path}"
|
||||
)
|
||||
|
|
@ -201,88 +250,99 @@ pkgs.testers.runNixOSTest {
|
|||
f"-r rest:http://{agent}:restpw-{agent}@127.0.0.1:${toString port}/{repo}/ {args}"
|
||||
)
|
||||
|
||||
# What swarm-controller writes: a user line and a restic password per agent
|
||||
# for the archive, a REST password and the restic password for the hive.
|
||||
def grant_agents(m):
|
||||
for agent in agents:
|
||||
put(m, f"secret/swarm/services/archive/users/{agent}", "htpasswd", sha_line(m, agent, f"restpw-{agent}"))
|
||||
put(m, f"secret/swarm/services/archive/restic/{agent}", "password", f"resticpw-{agent}")
|
||||
put(m, f"secret/swarm/hives/${hive}/archive/{agent}", "rest-password", f"restpw-{agent}")
|
||||
put(m, f"secret/swarm/hives/${hive}/archive/{agent}", "restic-password", f"resticpw-{agent}")
|
||||
m.succeed(
|
||||
f"mkdir -p ${agentRoot}/{agent}/state ${agentRoot}/{agent}/claude "
|
||||
f"${agentRoot}/{agent}/harness ${agentRoot}/{agent}/work"
|
||||
)
|
||||
m.succeed(f"echo {agent}-state > ${agentRoot}/{agent}/state/data")
|
||||
m.succeed(f"echo {agent}-work > ${agentRoot}/{agent}/work/scratch")
|
||||
|
||||
# What the archive promises, whichever backend holds the repositories.
|
||||
def archive_suite(m, tag):
|
||||
with subtest(f"{tag}: the server starts with no users and admits nobody"):
|
||||
m.succeed("test -e /run/swarm-archive-auth/htpasswd")
|
||||
assert status(m, "alpha", "alpha") == "401", status(m, "alpha", "alpha")
|
||||
|
||||
grant_agents(m)
|
||||
|
||||
with subtest(f"{tag}: the users file follows the store without a restart of the server"):
|
||||
pid = m.succeed("systemctl show -p MainPID --value swarm-archive").strip()
|
||||
m.succeed("systemctl start swarm-archive-users")
|
||||
assert m.succeed("systemctl show -p MainPID --value swarm-archive").strip() == pid
|
||||
assert status(m, "alpha", "alpha") in ("200", "404"), status(m, "alpha", "alpha")
|
||||
assert status(m, "beta", "beta") in ("200", "404"), status(m, "beta", "beta")
|
||||
# Wrong password: the control that the 200 above is a login, not an open door.
|
||||
m.succeed(f"test $(curl -s -o /dev/null -w '%{{http_code}}' -u alpha:wrong {url}/alpha/config) = 401")
|
||||
|
||||
with subtest(f"{tag}: the users file holds hashes, readable by the server only"):
|
||||
m.succeed("grep -q '{SHA}' /run/swarm-archive-auth/htpasswd")
|
||||
m.fail("grep -q restpw /run/swarm-archive-auth/htpasswd")
|
||||
m.succeed("test \"$(stat -c '%a %U:%G' /run/swarm-archive-auth/htpasswd)\" = '640 root:swarm-archive'")
|
||||
|
||||
with subtest(f"{tag}: the hive's backup puts each agent in its own repository, without work/"):
|
||||
m.succeed("systemctl start test-archive-backup")
|
||||
for agent in agents:
|
||||
snaps = m.succeed(restic(agent, agent, "snapshots --json"))
|
||||
assert snaps.count('"id"') == 1, snaps
|
||||
listing = m.succeed(restic(agent, agent, "ls latest"))
|
||||
assert f"/{agent}/state/data" in listing, listing
|
||||
assert "scratch" not in listing, listing
|
||||
|
||||
with subtest(f"{tag}: one agent's login cannot reach another's repository"):
|
||||
assert status(m, "beta", "alpha") == "403", status(m, "beta", "alpha")
|
||||
m.fail(restic("beta", "alpha", "snapshots"))
|
||||
assert status(m, "alpha", "alpha") == "200", status(m, "alpha", "alpha")
|
||||
assert status(m, "alpha", "beta") == "403", status(m, "alpha", "beta")
|
||||
|
||||
with subtest(f"{tag}: a client cannot delete or overwrite"):
|
||||
snap = m.succeed(restic("alpha", "alpha", "snapshots --json")).split('"id":"')[1].split('"')[0]
|
||||
# An old snapshot for the prune to drop. The backup is also the presence control: a lock, which a backup creates and removes, is deletable.
|
||||
m.succeed(restic("alpha", "alpha", "backup --host alpha --time '2000-01-01 00:00:00' ${agentRoot}/alpha/state"))
|
||||
assert status(m, "alpha", "alpha", f"snapshots/{snap}", "DELETE") == "403"
|
||||
assert status(m, "alpha", "alpha", "config", "POST") == "403"
|
||||
m.fail(restic("alpha", "alpha", "forget --keep-last 1 --prune"))
|
||||
assert m.succeed(restic("alpha", "alpha", "snapshots --json")).count('"id"') == 2
|
||||
|
||||
with subtest(f"{tag}: the server-side prune applies the retention"):
|
||||
m.succeed("systemctl start swarm-archive-prune")
|
||||
for agent in agents:
|
||||
assert m.succeed(restic(agent, agent, "snapshots --json")).count('"id"') == 1
|
||||
# The old snapshot went, the one from the hive's backup stayed.
|
||||
assert snap in m.succeed(restic("alpha", "alpha", "snapshots --json"))
|
||||
|
||||
with subtest(f"{tag}: a snapshot dated in the future cannot push the real ones out of the retention"):
|
||||
# restic counts --keep-within from the newest snapshot, and the client sets its time.
|
||||
m.succeed(restic("alpha", "alpha", "backup --host alpha --time '2100-01-01 00:00:00' ${agentRoot}/alpha/state"))
|
||||
assert m.succeed(restic("alpha", "alpha", "snapshots --json")).count('"id"') == 2
|
||||
m.succeed("systemctl start swarm-archive-prune")
|
||||
left = m.succeed(restic("alpha", "alpha", "snapshots --json"))
|
||||
assert left.count('"id"') == 1 and snap in left, left
|
||||
m.succeed("journalctl -u swarm-archive-prune | grep -q 'archive prune warning: alpha has snapshots dated after now'")
|
||||
# beta's repository had nothing forged, and says nothing.
|
||||
m.fail("journalctl -u swarm-archive-prune | grep -q 'archive prune warning: beta'")
|
||||
|
||||
with subtest(f"{tag}: a user removed from the store is refused without a restart"):
|
||||
m.succeed(f"rm -r {store}/secret/swarm/services/archive/users/beta")
|
||||
m.succeed("systemctl start swarm-archive-users")
|
||||
assert status(m, "beta", "beta") == "401", status(m, "beta", "beta")
|
||||
assert status(m, "alpha", "alpha") == "200", status(m, "alpha", "alpha")
|
||||
|
||||
start_all()
|
||||
machine.wait_for_unit("multi-user.target")
|
||||
machine.wait_for_open_port(${toString port})
|
||||
|
||||
with subtest("the server starts with no users and admits nobody"):
|
||||
machine.succeed("test -e /run/swarm-archive-auth/htpasswd")
|
||||
assert status("alpha", "alpha") == "401", status("alpha", "alpha")
|
||||
archive_suite(machine, "local")
|
||||
|
||||
# What swarm-controller writes: a user line and a restic password per agent
|
||||
# for the archive, a REST password and the restic password for the hive.
|
||||
for agent in agents:
|
||||
put(f"secret/swarm/services/archive/users/{agent}", "htpasswd", sha_line(agent, f"restpw-{agent}"))
|
||||
put(f"secret/swarm/services/archive/restic/{agent}", "password", f"resticpw-{agent}")
|
||||
put(f"secret/swarm/hives/${hive}/archive/{agent}", "rest-password", f"restpw-{agent}")
|
||||
put(f"secret/swarm/hives/${hive}/archive/{agent}", "restic-password", f"resticpw-{agent}")
|
||||
machine.succeed(
|
||||
f"mkdir -p ${agentRoot}/{agent}/state ${agentRoot}/{agent}/claude "
|
||||
f"${agentRoot}/{agent}/harness ${agentRoot}/{agent}/work"
|
||||
)
|
||||
machine.succeed(f"echo {agent}-state > ${agentRoot}/{agent}/state/data")
|
||||
machine.succeed(f"echo {agent}-work > ${agentRoot}/{agent}/work/scratch")
|
||||
|
||||
with subtest("the users file follows the store without a restart of the server"):
|
||||
pid = machine.succeed("systemctl show -p MainPID --value swarm-archive").strip()
|
||||
machine.succeed("systemctl start swarm-archive-users")
|
||||
assert machine.succeed("systemctl show -p MainPID --value swarm-archive").strip() == pid
|
||||
assert status("alpha", "alpha") in ("200", "404"), status("alpha", "alpha")
|
||||
assert status("beta", "beta") in ("200", "404"), status("beta", "beta")
|
||||
# Wrong password: the control that the 200 above is a login, not an open door.
|
||||
machine.succeed(f"test $(curl -s -o /dev/null -w '%{{http_code}}' -u alpha:wrong {url}/alpha/config) = 401")
|
||||
|
||||
with subtest("the users file holds hashes, readable by the server only"):
|
||||
machine.succeed("grep -q '{SHA}' /run/swarm-archive-auth/htpasswd")
|
||||
machine.fail("grep -q restpw /run/swarm-archive-auth/htpasswd")
|
||||
machine.succeed("test \"$(stat -c '%a %U:%G' /run/swarm-archive-auth/htpasswd)\" = '640 root:swarm-archive'")
|
||||
|
||||
with subtest("the hive's backup puts each agent in its own repository, without work/"):
|
||||
machine.succeed("systemctl start test-archive-backup")
|
||||
for agent in agents:
|
||||
snaps = machine.succeed(restic(agent, agent, "snapshots --json"))
|
||||
assert snaps.count('"id"') == 1, snaps
|
||||
listing = machine.succeed(restic(agent, agent, "ls latest"))
|
||||
assert f"/{agent}/state/data" in listing, listing
|
||||
assert "scratch" not in listing, listing
|
||||
|
||||
with subtest("one agent's login cannot reach another's repository"):
|
||||
assert status("beta", "alpha") == "403", status("beta", "alpha")
|
||||
machine.fail(restic("beta", "alpha", "snapshots"))
|
||||
assert status("alpha", "alpha") == "200", status("alpha", "alpha")
|
||||
assert status("alpha", "beta") == "403", status("alpha", "beta")
|
||||
|
||||
with subtest("a client cannot delete or overwrite"):
|
||||
snap = machine.succeed(restic("alpha", "alpha", "snapshots --json")).split('"id":"')[1].split('"')[0]
|
||||
# An old snapshot for the prune to drop. The backup is also the presence control: a lock, which a backup creates and removes, is deletable.
|
||||
machine.succeed(restic("alpha", "alpha", "backup --host alpha --time '2000-01-01 00:00:00' ${agentRoot}/alpha/state"))
|
||||
assert status("alpha", "alpha", f"snapshots/{snap}", "DELETE") == "403"
|
||||
assert status("alpha", "alpha", "config", "POST") == "403"
|
||||
machine.fail(restic("alpha", "alpha", "forget --keep-last 1 --prune"))
|
||||
assert machine.succeed(restic("alpha", "alpha", "snapshots --json")).count('"id"') == 2
|
||||
|
||||
with subtest("the server-side prune applies the retention"):
|
||||
machine.succeed("systemctl start swarm-archive-prune")
|
||||
for agent in agents:
|
||||
assert machine.succeed(restic(agent, agent, "snapshots --json")).count('"id"') == 1
|
||||
# The old snapshot went, the one from the hive's backup stayed.
|
||||
assert snap in machine.succeed(restic("alpha", "alpha", "snapshots --json"))
|
||||
|
||||
with subtest("a snapshot dated in the future cannot push the real ones out of the retention"):
|
||||
# restic counts --keep-within from the newest snapshot, and the client sets its time.
|
||||
machine.succeed(restic("alpha", "alpha", "backup --host alpha --time '2100-01-01 00:00:00' ${agentRoot}/alpha/state"))
|
||||
assert machine.succeed(restic("alpha", "alpha", "snapshots --json")).count('"id"') == 2
|
||||
machine.succeed("systemctl start swarm-archive-prune")
|
||||
left = machine.succeed(restic("alpha", "alpha", "snapshots --json"))
|
||||
assert left.count('"id"') == 1 and snap in left, left
|
||||
machine.succeed("journalctl -u swarm-archive-prune | grep -q 'archive prune warning: alpha has snapshots dated after now'")
|
||||
# beta's repository had nothing forged, and says nothing.
|
||||
machine.fail("journalctl -u swarm-archive-prune | grep -q 'archive prune warning: beta'")
|
||||
|
||||
with subtest("a user removed from the store is refused without a restart"):
|
||||
machine.succeed(f"rm -r {store}/secret/swarm/services/archive/users/beta")
|
||||
machine.succeed("systemctl start swarm-archive-users")
|
||||
assert status("beta", "beta") == "401", status("beta", "beta")
|
||||
assert status("alpha", "alpha") == "200", status("alpha", "alpha")
|
||||
with subtest("the repositories are under the data directory"):
|
||||
machine.succeed("test -d /var/lib/swarm-archive/repos/alpha")
|
||||
|
||||
with subtest("the archive announces itself once, so swarm-controller starts granting"):
|
||||
machine.succeed(f"test \"$(cat {store}/secret/swarm/services/archive/ready/ready)\" = 1")
|
||||
|
|
@ -293,5 +353,45 @@ pkgs.testers.runNixOSTest {
|
|||
machine.succeed("systemctl start test-archive-backup")
|
||||
machine.succeed("journalctl -u test-archive-backup | grep -q 'granted this hive no archive credentials'")
|
||||
machine.fail("journalctl -u test-archive-backup | grep -q 'archive backup failed'")
|
||||
|
||||
# The same archive on the second node, with its repositories on the fake S3.
|
||||
s3archive.wait_for_unit("fake-s3.service")
|
||||
s3_key_id = s3archive.succeed("cat ${s3KeyDir}/id").strip()
|
||||
s3_secret_key = s3archive.succeed("cat ${s3KeyDir}/secret").strip()
|
||||
|
||||
with subtest("s3: without the key in the store the server does not start"):
|
||||
s3archive.wait_until_succeeds("journalctl -u swarm-archive | grep -q 'No value found'")
|
||||
s3archive.fail("systemctl is-active --quiet swarm-archive")
|
||||
s3archive.fail(f"curl -sf --max-time 2 -o /dev/null {url}/")
|
||||
|
||||
with subtest("s3: the server starts once the store holds the key"):
|
||||
put(s3archive, "secret/swarm/services/archive/s3/access-key-id", "value", s3_key_id)
|
||||
put(s3archive, "secret/swarm/services/archive/s3/secret-access-key", "value", s3_secret_key)
|
||||
s3archive.wait_for_open_port(${toString port})
|
||||
|
||||
archive_suite(s3archive, "s3")
|
||||
|
||||
with subtest("s3: the repositories are in the bucket and not on this host"):
|
||||
for agent in agents:
|
||||
s3archive.succeed(f"test -f ${s3Root}/${s3Bucket}/pfx/{agent}/config")
|
||||
s3archive.succeed("find ${s3Root}/${s3Bucket}/pfx/alpha/snapshots -type f | grep -q .")
|
||||
s3archive.succeed("find ${s3Root}/${s3Bucket}/pfx/alpha/data -type f | grep -q .")
|
||||
s3archive.fail("test -e /var/lib/swarm-archive/repos")
|
||||
|
||||
with subtest("s3: the key is in rclone's environment and in no unit"):
|
||||
pid = s3archive.succeed("systemctl show -p MainPID --value swarm-archive").strip()
|
||||
env = s3archive.succeed(f"tr '\\0' '\\n' < /proc/{pid}/environ")
|
||||
assert f"RCLONE_S3_ACCESS_KEY_ID={s3_key_id}" in env, env
|
||||
assert f"RCLONE_S3_SECRET_ACCESS_KEY={s3_secret_key}" in env, env
|
||||
# The connection settings are in the unit text, the control that the search
|
||||
# for the key below looks at the units.
|
||||
units = "swarm-archive swarm-archive-prune swarm-archive-users"
|
||||
s3archive.succeed(f"systemctl cat {units} | grep -q RCLONE_S3_ENDPOINT")
|
||||
s3archive.fail(f"systemctl cat {units} | grep -q {s3_key_id}")
|
||||
s3archive.fail(f"systemctl cat {units} | grep -q {s3_secret_key}")
|
||||
wrapper = s3archive.succeed("systemctl show -p ExecStart --value swarm-archive | grep -o '/nix/store/[^ ;]*swarm-archive-serve' | head -1").strip()
|
||||
s3archive.succeed(f"grep -q 'kv get' {wrapper}")
|
||||
s3archive.fail(f"grep -q {s3_key_id} {wrapper}")
|
||||
s3archive.fail(f"grep -rq {s3_key_id} /run/swarm-archive-auth /var/cache/swarm-archive-prune")
|
||||
'';
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue