diff --git a/docs/2-handbook/swarm/archive.md b/docs/2-handbook/swarm/archive.md index 72779b38b..2d0a24725 100644 --- a/docs/2-handbook/swarm/archive.md +++ b/docs/2-handbook/swarm/archive.md @@ -33,8 +33,9 @@ as for the binary cache. Each agent has its own [restic](https://restic.net) repository at `https://archive.//`, under `deploy.archive.dataDir` -(default `/var/lib/swarm-archive`) on that host. restic encrypts it with a -password of its own for that agent. +(default `/var/lib/swarm-archive`) on that host, or in a bucket of an +[external S3](#external-s3). restic encrypts it with a password of its own for +that agent. Two properties protect the backups from a compromised hive: @@ -49,6 +50,54 @@ Two properties protect the backups from a compromised hive: - **Append-only.** A client can add snapshots and nothing else: no delete, no overwrite. The archive host removes old snapshots. +## External S3 + +By default the repositories sit on the archive host's disk. With +`deploy.archive.backend = "s3"` they sit in a bucket of an S3 service you run +or rent instead, and the host keeps nothing. Hives don't change: they still +talk to `archive.` with their per-agent logins, and the archive +is still append-only. Only the archive holds the S3 key, so a compromised hive +can't reach the bucket directly. + +| option | default | meaning | +| ---------------------------- | ------- | ----------------------------------------------------------------- | +| `deploy.archive.backend` | `local` | `local` or `s3` | +| `deploy.archive.s3.endpoint` | `null` | the service's URL, scheme included; required | +| `deploy.archive.s3.bucket` | `null` | the bucket; required, and it must exist | +| `deploy.archive.s3.prefix` | `""` | a path in the bucket; the repositories sit at `//` | +| `deploy.archive.s3.region` | `null` | the bucket's region, for services that need one | +| `deploy.archive.s3.provider` | `Other` | rclone's `--s3-provider`, for example `AWS` | + +Evaluation fails if `backend` is `s3` and the endpoint or the bucket is unset. + +The service needs one bucket and one access key that can list, read, write +and delete objects in it. Scope the key to the bucket. The archive never +creates a bucket, so the key needs no permission to. The prune removes old +data, which takes delete rights. Hives can't delete: the archive refuses it +before the request reaches the bucket. + +The key lives in the secret store, like every other archive secret. Write it +before you set `backend = "s3"`; the server doesn't start without it. Read it +from a file or the terminal so it doesn't land in your shell history: + +```sh +bao kv put secret/swarm/services/archive/s3/access-key-id value=- +bao kv put secret/swarm/services/archive/s3/secret-access-key value=- +``` + +Each command reads the value from standard input. The store's UI writes the +same two leaves, field `value`. The archive's host reads them as it starts and +passes them to rclone in its environment. They never reach a file there, so +a changed key takes effect at the next restart of `swarm-archive`. The daily +prune reads them on every run. If the server can't read them, it fails to +start and systemd retries every five seconds until the store answers. + +Changing `backend` doesn't move the data. Before you switch an archive that +holds backups, copy them with `rclone copy /repos :s3:/` +using a rclone configured for the bucket, or start with empty repositories and +let the next hourly runs fill them. The old directory stays on the host until +you remove it. + ## How long it stays The archive host prunes every repository once a day. A snapshot survives if @@ -128,10 +177,14 @@ hives with the unit log that they have no credentials and succeed until it updates. The archive host admits only the users the controller wrote, so until then it admits nobody. +Where the repositories live is a setting of the archive host alone. Hives and `swarm-controller` +behave the same with either. + ## What lives where Everything secret lives in the swarm secret store, and the archive keeps none on disk. The archive host renders its users file (password hashes only) into -`/run/swarm-archive-auth/`, which is tmpfs. Hives read their restic password -when restic asks for it. The paths and who writes and reads them are in the +`/run/swarm-archive-auth/`, which is tmpfs. With an external S3, it reads the +S3 key from the store when a process starts and passes it to rclone in the +environment. Hives read their restic password when restic asks for it. The paths and who writes and reads them are in the [credentials reference](credentials-reference.md). diff --git a/docs/2-handbook/swarm/credentials-reference.md b/docs/2-handbook/swarm/credentials-reference.md index ec55d7b81..522ae23cc 100644 --- a/docs/2-handbook/swarm/credentials-reference.md +++ b/docs/2-handbook/swarm/credentials-reference.md @@ -61,6 +61,8 @@ the rest of the cell says how. | `swarm/services/archive/users/` (field `htpasswd`) | `swarm-controller`, with the REST password above: its `{SHA}` hash, never the password; deleted when no hive places the agent | `swarm-archive-users` on the archive's host, which renders the users file into `/run` | ✅ rewritten with the password | ✅ re-read every 2 minutes; the server notices the new file without a restart | | `swarm/services/archive/restic/` (field `password`) | `swarm-controller`, once per agent, on its first placement; never replaced and never deleted, because it opens every backup of the agent | `swarm-archive-prune` on the archive's host, through `RESTIC_PASSWORD_COMMAND` | ❌ a new password would not open the existing repository | ✅ read at every prune | | `swarm/services/archive/ready` | `swarm-archive-users` on the archive's host, once, when none is stored; `swarm-controller` writes no archive credentials before it exists | `swarm-controller`, every 5 minutes | ❌ never rewritten | ✅ read every 5 minutes | +| `swarm/services/archive/s3/access-key-id` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive` and `swarm-archive-prune` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune | +| `swarm/services/archive/s3/secret-access-key` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive` and `swarm-archive-prune` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune | | `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass | | _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated | @@ -185,7 +187,7 @@ login minter's, the forge host's and Grafana's, which reach more than one: | `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services//oidc/client` | | `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services//oidc/client` | | `swarm-bao-attic-secret` and `swarm-bao-attic-tokens` | `atticClientCertFile` / `atticClientKeyFile` | creates and reads `swarm/services/attic/jwt-secret`, `swarm/services/attic/push-token` and `swarm/services/attic/pull-token` | -| `swarm-archive-users` and `swarm-archive-prune` | `archiveClientCertFile` / `archiveClientKeyFile` | reads `swarm/services/archive/users/*` and `swarm/services/archive/restic/*`, lists `swarm/services/archive/users/` | +| `swarm-archive-users`, `swarm-archive-prune` and, with an S3 bucket, `swarm-archive` | `archiveClientCertFile` / `archiveClientKeyFile` | reads `swarm/services/archive/users/*`, `swarm/services/archive/restic/*` and `swarm/services/archive/s3/*`, lists `swarm/services/archive/users/` | | `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services//oidc/client` | | `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services//oidc/client` and `swarm/services//oidc/client` | diff --git a/nix/host-modules/lib/archive-server.nix b/nix/host-modules/lib/archive-server.nix index a052bc747..d05dd53e8 100644 --- a/nix/host-modules/lib/archive-server.nix +++ b/nix/host-modules/lib/archive-server.nix @@ -5,16 +5,24 @@ # (nix/vm-tests/swarm-archive.nix) runs the same scripts against a stand-in # `bao`. # -# Both scripts use only `bao login -method=cert -token-only`, +# All scripts use only `bao login -method=cert -token-only`, # `bao kv list -format=json` and `bao kv get -field=…`, and need `bao`, `jq`, # `coreutils`, `gnugrep` on the unit's `path` and the `BAO_*` client set. +# +# The repositories live in a directory (`reposDir`) or in an S3 bucket (`s3`). +# Only the pieces that name the backend differ: the remote rclone serves, how +# the prune lists and addresses the repositories, and the S3 key, which is read +# from the secret store when a process starts. { lib }: { rclone, restic, - # Where the repositories live: `//`. The only place that - # names the backend; `rclone serve restic` takes any remote here. - reposDir, + # Local backend: the repositories live at `//`. Null with `s3`. + reposDir ? null, + # S3 backend: `{ bucket, prefix, endpoint, region, provider }`, the + # repositories at `///`; `region` may be null and + # `prefix` empty. Null with `reposDir`. + s3 ? null, # The address rclone listens on. addr, # The users file rclone checks logins against, its group and the user the @@ -32,6 +40,8 @@ let usersPrefix = "secret/swarm/services/archive/users"; resticPrefix = "secret/swarm/services/archive/restic"; readyPath = "secret/swarm/services/archive/ready"; + s3KeyIdPath = "secret/swarm/services/archive/s3/access-key-id"; + s3SecretKeyPath = "secret/swarm/services/archive/s3/secret-access-key"; # Slack for the clocks of the hives that stamp snapshots. futureSkewSeconds = 3600; @@ -42,13 +52,48 @@ let retention.keepWithin != null ) "--keep-within ${lib.escapeShellArg retention.keepWithin}" ); -in -{ - inherit usersPrefix resticPrefix; - # Append-only: a client cannot delete or overwrite a snapshot, a pack or the - # index, so a hive that is compromised cannot erase history. Private repos: - # a login reaches `//` and nothing else. + # An rclone connection string, so no rclone config exists; the connection + # settings are `environment` below. + s3Remote = ":s3:${s3.bucket}${lib.optionalString (s3.prefix != "") "/${s3.prefix}"}"; + + # What rclone is told to serve. + backendArg = if s3 == null then reposDir else s3Remote; + + # Reads the S3 key from the store into rclone's environment. Needs the + # caller's store login. + s3Credentials = '' + RCLONE_S3_ACCESS_KEY_ID="$(timeout 15 bao kv get -field=value ${lib.escapeShellArg s3KeyIdPath})" + RCLONE_S3_SECRET_ACCESS_KEY="$(timeout 15 bao kv get -field=value ${lib.escapeShellArg s3SecretKeyPath})" + export RCLONE_S3_ACCESS_KEY_ID RCLONE_S3_SECRET_ACCESS_KEY + ''; + + # Prints the name of every agent that has a repository, one per line. + listAgents = + if s3 == null then + '' + for repo in ${lib.escapeShellArg reposDir}/*/; do + [ -e "$repo/config" ] || continue + basename -- "$repo" + done + '' + else + '' + rclone lsf --dirs-only ${lib.escapeShellArg "${s3Remote}/"} | while read -r dir; do + dir="''${dir%/}" + [ -n "$(rclone lsf ${lib.escapeShellArg s3Remote}/"$dir"/config)" ] || continue + printf '%s\n' "$dir" + done + ''; + + # restic's name for the repository of `$agent`: the directory, or the rclone + # backend over the same remote the server serves. + repoOf = + if s3 == null then + ''${lib.escapeShellArg reposDir}/"$agent"/'' + else + ''${lib.escapeShellArg "rclone:${s3Remote}/"}"$agent"''; + serverCommand = lib.escapeShellArgs [ "${rclone}/bin/rclone" "serve" @@ -59,8 +104,41 @@ in "--private-repos" "--htpasswd" authFile - reposDir + backendArg ]; +in +{ + inherit usersPrefix resticPrefix; + + # The S3 remote's connection settings, everything but the key. Empty for the + # local backend. + environment = lib.optionalAttrs (s3 != null) ( + { + RCLONE_S3_PROVIDER = s3.provider; + RCLONE_S3_ENDPOINT = s3.endpoint; + # A key scoped to the bucket cannot create buckets. + RCLONE_S3_NO_CHECK_BUCKET = "true"; + RCLONE_S3_ENV_AUTH = "false"; + } + // lib.optionalAttrs (s3.region != null) { RCLONE_S3_REGION = s3.region; } + ); + + # Append-only: a client cannot delete or overwrite a snapshot, a pack or the + # index, so a hive that is compromised cannot erase history. Private repos: + # a login reaches `//` and nothing else. + inherit serverCommand; + + # Start of the S3 server: logs in with the unit's store identity, puts the key + # in rclone's environment and becomes rclone. The key is never written out. A + # store that cannot answer fails the start, and the unit's restart retries. + s3ServerScript = '' + set -euo pipefail + BAO_TOKEN="$(timeout 15 bao login -method=cert -token-only)" + export BAO_TOKEN + ${s3Credentials} + unset BAO_TOKEN + exec ${serverCommand} + ''; # Renders the users file from `usersPrefix`, one leaf per agent holding that # agent's htpasswd line. The file is written only when the content changed, @@ -171,15 +249,21 @@ in exit 1 fi + ${lib.optionalString (s3 != null) s3Credentials} + list_agents() { + ${listAgents} + } + # Under `set -e` a failed listing ends the run; inside the `for` it would not. + agents="$(list_agents)" + failed=0 - for repo in ${lib.escapeShellArg reposDir}/*/; do - [ -e "$repo/config" ] || continue - agent="$(basename -- "$repo")" + for agent in $agents; do if [[ ! "$agent" =~ ^[A-Za-z0-9_-]+$ ]]; then echo "archive prune failed: $agent is not an agent name" >&2 failed=1 continue fi + repo=${repoOf} export RESTIC_PASSWORD_COMMAND="bao kv get -field=password ${resticPrefix}/$agent" if forget_future "$repo" "$agent" \ && ${restic}/bin/restic -r "$repo" --retry-lock 10m forget --group-by host ${keepFlags} --prune; then diff --git a/nix/host-modules/swarm-archive.nix b/nix/host-modules/swarm-archive.nix index fb896fe1e..457350579 100644 --- a/nix/host-modules/swarm-archive.nix +++ b/nix/host-modules/swarm-archive.nix @@ -23,11 +23,11 @@ # therefore decrypt every backup, which is the cost of pruning where clients # cannot delete. # -# Both units log in as the archive's own store principal (a leaf of the store's +# The units log in as the archive's own store principal (a leaf of the store's # CA, minted by ./glue-bao-tls.nix). Plain host units rather than a container: -# no secret crosses a mount, and the backend is one string -# (`reposDir`, in ./lib/archive-server.nix) that names where the repositories -# are. +# no secret crosses a mount. The backend is one argument (`reposDir` or `s3`, in +# ./lib/archive-server.nix); with `s3` the server also reads the S3 key from +# `swarm/services/archive/s3/…` as it starts and keeps it in rclone's environment. { pkgs, lib, @@ -51,17 +51,40 @@ let authFile = "${authDir}/htpasswd"; reposDir = "${cfg.dataDir}/repos"; - server = import ./lib/archive-server.nix { inherit lib; } { - inherit (cfg) retention; - inherit authFile reposDir; - rclone = cfg.package; - restic = cfg.resticPackage; - addr = "127.0.0.1:${toString cfg.port}"; - authGroup = user; - }; + useS3 = cfg.backend == "s3"; + + server = import ./lib/archive-server.nix { inherit lib; } ( + { + inherit (cfg) retention; + inherit authFile; + rclone = cfg.package; + restic = cfg.resticPackage; + addr = "127.0.0.1:${toString cfg.port}"; + authGroup = user; + } + // (if useS3 then { inherit (cfg) s3; } else { inherit reposDir; }) + ); storeRetry = import ./lib/store-retry.nix { }; + # A unit that runs as the repositories' owner gets the store identity as + # credentials, which only that user can read. + ownerStoreCredentials = [ + "bao-cert:${baoDeploy.archiveClientCertFile}" + "bao-key:${baoDeploy.archiveClientKeyFile}" + ] + ++ lib.optional (baoDeploy.serverCaFile != null) "bao-ca:${baoDeploy.serverCaFile}"; + ownerStoreEnvironment = { + BAO_ADDR = storeEnvironment.BAO_ADDR; + BAO_CLIENT_CERT = "%d/bao-cert"; + BAO_CLIENT_KEY = "%d/bao-key"; + } + // lib.optionalAttrs (baoDeploy.serverCaFile != null) { + BAO_CACERT = "%d/bao-ca"; + }; + + serveScript = pkgs.writeShellScript "swarm-archive-serve" server.s3ServerScript; + haveClientIdentity = baoDeploy.archiveClientCertFile != null && baoDeploy.archiveClientKeyFile != null; @@ -101,6 +124,72 @@ in description = "The restic package the prune unit runs."; }; + backend = lib.mkOption { + type = lib.types.enum [ + "local" + "s3" + ]; + default = "local"; + description = '' + Where the repositories are kept: `local` is a directory on this host + ({option}`services.hyperhive.deploy.archive.dataDir`), `s3` a bucket of an + external S3 service (`services.hyperhive.deploy.archive.s3.*`). + Hives always talk to this archive, whatever the backend, and never see the + S3 key. The archive reads the key from the secret store, in the leaves + `swarm/services/archive/s3/access-key-id` and `…/secret-access-key` + (field `value`), which you write. + + Switching does not move the repositories already stored. Copy them first + (`rclone copy /repos :s3:/`) or start with + empty repositories. + ''; + }; + + s3 = { + endpoint = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "https://s3.eu-central-1.amazonaws.com"; + description = "URL of the S3 service, scheme included. Required with `backend = \"s3\"`."; + }; + + bucket = lib.mkOption { + type = lib.types.nullOr (lib.types.strMatching "[A-Za-z0-9._-]+"); + default = null; + description = '' + The bucket that holds the repositories. Required with `backend = "s3"`. + It must exist already: the archive creates none. + ''; + }; + + prefix = lib.mkOption { + type = lib.types.strMatching "([A-Za-z0-9._-]+(/[A-Za-z0-9._-]+)*)?"; + default = ""; + example = "archive"; + description = '' + A path in the bucket, without leading or trailing slash. The + repositories sit at `//`; empty puts them at the top. + ''; + }; + + region = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "eu-central-1"; + description = "The bucket's region, for services that require one. `null` leaves rclone's default."; + }; + + provider = lib.mkOption { + type = lib.types.str; + default = "Other"; + example = "AWS"; + description = '' + rclone's `--s3-provider` value, which selects the service's quirks. + `Other` suits a generic S3-compatible service. + ''; + }; + }; + port = lib.mkOption { type = lib.types.port; default = 8911; @@ -114,8 +203,8 @@ in type = lib.types.path; default = "/var/lib/swarm-archive"; description = '' - Where the repositories live, under `repos//`. Holds restic - ciphertext: every backup of every agent, so size it for that. + Where the repositories live with `backend = "local"`, under `repos//`. + Holds restic ciphertext: every backup of every agent, so size it for that. ''; }; @@ -172,6 +261,14 @@ in host's /var/lib/swarm-bao-pki/ and name them in those two options. ''; } + { + assertion = !useS3 || (cfg.s3.endpoint != null && cfg.s3.bucket != null); + message = '' + services.hyperhive.deploy.archive.backend = "s3" requires both + services.hyperhive.deploy.archive.s3.endpoint and + services.hyperhive.deploy.archive.s3.bucket. + ''; + } ]; users.users.${user} = { @@ -212,9 +309,11 @@ in ]; systemd.tmpfiles.rules = [ + "d ${authDir} 0750 root ${user} -" + ] + ++ lib.optionals (!useS3) [ "d ${cfg.dataDir} 0750 ${user} ${user} -" "d ${reposDir} 0750 ${user} ${user} -" - "d ${authDir} 0750 root ${user} -" ]; systemd.services.swarm-archive = { @@ -227,9 +326,16 @@ in "systemd-tmpfiles-setup.service" ]; # rclone writes no config, and reads none. - environment.RCLONE_CONFIG = "/dev/null"; + environment = { + RCLONE_CONFIG = "/dev/null"; + } + // lib.optionalAttrs useS3 (ownerStoreEnvironment // server.environment); + path = lib.optionals useS3 [ + baoDeploy.package + pkgs.coreutils + ]; serviceConfig = { - ExecStart = server.serverCommand; + ExecStart = if useS3 then serveScript else server.serverCommand; User = user; Group = user; Restart = "on-failure"; @@ -246,9 +352,10 @@ in "AF_INET" "AF_INET6" ]; - ReadWritePaths = [ reposDir ]; ReadOnlyPaths = [ authDir ]; - }; + } + // lib.optionalAttrs useS3 { LoadCredential = ownerStoreCredentials; } + // lib.optionalAttrs (!useS3) { ReadWritePaths = [ reposDir ]; }; }; # The users file, from the store. A failed run leaves the file as it is; @@ -303,37 +410,30 @@ in pkgs.coreutils pkgs.gnugrep pkgs.jq - ]; - # The unit runs as the repositories' owner, so the store identity comes - # in as credentials, which only it can read. - environment = { - BAO_ADDR = storeEnvironment.BAO_ADDR; - BAO_CLIENT_CERT = "%d/bao-cert"; - BAO_CLIENT_KEY = "%d/bao-key"; - RESTIC_CACHE_DIR = "%C/swarm-archive-prune"; - } - // lib.optionalAttrs (baoDeploy.serverCaFile != null) { - BAO_CACERT = "%d/bao-ca"; - }; + ] + # restic's rclone backend starts rclone. + ++ lib.optional useS3 cfg.package; + environment = + ownerStoreEnvironment + // { + RESTIC_CACHE_DIR = "%C/swarm-archive-prune"; + } + // lib.optionalAttrs useS3 ({ RCLONE_CONFIG = "/dev/null"; } // server.environment); serviceConfig = { Type = "oneshot"; User = user; Group = user; SyslogIdentifier = "swarm-archive-prune"; CacheDirectory = "swarm-archive-prune"; - LoadCredential = [ - "bao-cert:${baoDeploy.archiveClientCertFile}" - "bao-key:${baoDeploy.archiveClientKeyFile}" - ] - ++ lib.optional (baoDeploy.serverCaFile != null) "bao-ca:${baoDeploy.serverCaFile}"; + LoadCredential = ownerStoreCredentials; NoNewPrivileges = true; ProtectSystem = "strict"; ProtectHome = true; PrivateTmp = true; PrivateDevices = true; - ReadWritePaths = [ reposDir ]; TimeoutStartSec = "6h"; - }; + } + // lib.optionalAttrs (!useS3) { ReadWritePaths = [ reposDir ]; }; script = server.pruneScript; }; diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index 6d226a3d1..0963ec0b5 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -1394,7 +1394,8 @@ let # The backup archive: the users file is rendered from one leaf per agent, and # the prune reads each agent's restic password. Read and list only; the # controller writes both. It also creates the marker that tells the - # controller an archive exists. + # controller an archive exists. With an external S3 backend it reads the + # provider key, which the operator writes. archiveReaders = [ { name = "swarm-archive"; @@ -1402,6 +1403,7 @@ let policyText = readStanza "${credentialMountPath}/data/swarm/services/archive/users/*" + readStanza "${credentialMountPath}/data/swarm/services/archive/restic/*" + + readStanza "${credentialMountPath}/data/swarm/services/archive/s3/*" + listStanza "${credentialMountPath}/metadata/swarm/services/archive/users/" + createReadStanza "${credentialMountPath}/data/swarm/services/archive/ready"; } @@ -2685,7 +2687,7 @@ in description = '' Subject the store's `swarm-archive` cert-auth role accepts: the identity the backup archive's host units present. Its grant is read on - `swarm/services/archive/users/*` and `restic/*`, and list on `users`. + `swarm/services/archive/users/*`, `restic/*` and `s3/*`, and list on `users`. ⚠️ Reserved as a hive name by ./swarm.nix, like its siblings. ''; diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index 69598b81e..8e722d434 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -2201,6 +2201,9 @@ let in (grantFor grants "secret/data/swarm/services/archive/users/atlas").caps == [ "read" ] && (grantFor grants "secret/data/swarm/services/archive/restic/atlas").caps == [ "read" ] + && (grantFor grants "secret/data/swarm/services/archive/s3/access-key-id").caps == [ "read" ] + && (grantFor grants "secret/data/swarm/services/archive/s3/secret-access-key").caps == [ "read" ] + && grantFor c "secret/data/swarm/services/archive/s3/access-key-id" == null && (grantFor grants "secret/metadata/swarm/services/archive/users/").caps == [ "list" ] && (grantFor grants "secret/data/swarm/services/archive/ready").caps == [ @@ -2209,7 +2212,7 @@ let ] && (grantFor c "secret/data/swarm/services/archive/ready").caps == [ "read" ] && grantFor grants "secret/data/swarm/hives/h1/archive/atlas" == null - && lib.length grants == 4 + && lib.length grants == 5 && lib.hasInfix "allowed_common_names=swarm-archive" s && lib.hasInfix "token_policies=swarm-archive" s && lib.elem "create" (grantFor c "secret/data/swarm/services/archive/users/atlas").caps diff --git a/nix/module-eval/builder.nix b/nix/module-eval/builder.nix index 0ee138621..2d3dd02b8 100644 --- a/nix/module-eval/builder.nix +++ b/nix/module-eval/builder.nix @@ -46,6 +46,8 @@ let deploy.builder.enable = true; swarm.attic.domain = "cache.example.org"; swarm.attic.cache = null; + deploy.archive.enable = true; + deploy.archive.backend = "s3"; }; refusedHalfCache = m: @@ -109,6 +111,13 @@ let deploy.attic.enable = lib.mkForce false; swarm.attic.domain = "cache.example.org"; swarm.attic.cache = "grid"; + deploy.archive.backend = "s3"; + deploy.archive.s3 = { + endpoint = "https://s3.example.org"; + region = "eu-central-1"; + bucket = "bkt"; + prefix = "archive"; + }; }; atticWarnings = m: @@ -600,6 +609,102 @@ let && !(bare.systemd.services ? swarm-archive) && !(bare.systemd.services ? swarm-archive-users); } + { + # The default is the directory backend, and its unit is the one that ran + # before the S3 option: the literal command line is the control. + name = "the archive's default backend is the local directory and renders the unchanged server unit"; + ok = + let + a = atticHere.systemd.services.swarm-archive; + prune = atticHere.systemd.services.swarm-archive-prune; + cfg = atticHere.services.hyperhive.deploy.archive; + in + cfg.backend == "local" + && + a.serviceConfig.ExecStart + == "${cfg.package}/bin/rclone serve restic --addr 127.0.0.1:8911 --append-only --private-repos --htpasswd /run/swarm-archive-auth/htpasswd /var/lib/swarm-archive/repos" + && a.environment.RCLONE_CONFIG == "/dev/null" + && !(lib.any (n: lib.hasPrefix "RCLONE_S3" n) (lib.attrNames a.environment)) + && a.serviceConfig.ReadWritePaths == [ "/var/lib/swarm-archive/repos" ] + && !(a.serviceConfig ? LoadCredential) + && prune.serviceConfig.ReadWritePaths == [ "/var/lib/swarm-archive/repos" ] + && lib.hasInfix "/var/lib/swarm-archive/repos" prune.script + && !(lib.hasInfix "rclone" prune.script) + && lib.elem "d /var/lib/swarm-archive/repos 0750 swarm-archive swarm-archive -" atticHere.systemd.tmpfiles.rules; + } + { + # The key is read as the server starts and lives in rclone's environment: + # the unit carries the connection settings and the store identity, never + # the key, and the prune reaches the bucket through restic's rclone + # backend. The local case above is the control for each absence. + name = "the archive on an external S3 serves and prunes the bucket, reads its key from the store at start and keeps it out of the unit"; + ok = + let + a = atticExternal.systemd.services.swarm-archive; + prune = atticExternal.systemd.services.swarm-archive-prune; + cfg = atticExternal.services.hyperhive.deploy.archive; + server = import ../host-modules/lib/archive-server.nix { inherit lib; } { + rclone = pkgs.rclone; + restic = pkgs.restic; + addr = "127.0.0.1:8911"; + authFile = "/run/swarm-archive-auth/htpasswd"; + authGroup = "swarm-archive"; + retention = { + keepWithin = "3d"; + keepLast = 3; + }; + s3 = cfg.s3; + }; + unitText = builtins.toJSON [ + a.environment + a.serviceConfig + prune.environment + prune.serviceConfig + ]; + in + cfg.backend == "s3" + && lib.hasInfix "swarm-archive-serve" a.serviceConfig.ExecStart + && a.environment.RCLONE_S3_ENDPOINT == "https://s3.example.org" + && a.environment.RCLONE_S3_REGION == "eu-central-1" + && a.environment.RCLONE_S3_PROVIDER == "Other" + && a.environment.RCLONE_S3_NO_CHECK_BUCKET == "true" + && a.environment.RCLONE_S3_ENV_AUTH == "false" + && a.environment.RCLONE_CONFIG == "/dev/null" + && a.environment.BAO_CLIENT_CERT == "%d/bao-cert" + && lib.elem "bao-cert:/var/lib/swarm-bao-pki/archive.pem" a.serviceConfig.LoadCredential + && !(a.serviceConfig ? ReadWritePaths) + && !(prune.serviceConfig ? ReadWritePaths) + && prune.environment.RCLONE_S3_ENDPOINT == "https://s3.example.org" + && !(lib.elem "d /var/lib/swarm-archive/repos 0750 swarm-archive swarm-archive -" atticExternal.systemd.tmpfiles.rules) + && lib.hasInfix "RCLONE_S3_ENDPOINT" unitText + && !(lib.hasInfix "ACCESS_KEY" unitText) + && lib.hasInfix "--append-only" server.serverCommand + && lib.hasInfix "--private-repos" server.serverCommand + && lib.hasSuffix " :s3:bkt/archive" server.serverCommand + && lib.hasInfix "bao kv get -field=value secret/swarm/services/archive/s3/access-key-id" server.s3ServerScript + && lib.hasInfix "bao kv get -field=value secret/swarm/services/archive/s3/secret-access-key" server.s3ServerScript + && lib.hasInfix "\nexec /nix/store/" server.s3ServerScript + && lib.hasInfix "/bin/rclone serve restic" server.s3ServerScript + && lib.hasInfix "rclone lsf --dirs-only :s3:bkt/archive/" server.pruneScript + && lib.hasInfix "repo=rclone::s3:bkt/archive/\"$agent\"" server.pruneScript + && lib.hasInfix "RCLONE_S3_SECRET_ACCESS_KEY" server.pruneScript + && !(lib.hasInfix "/var/lib/swarm-archive" server.pruneScript); + } + { + # `halfCache` hosts the archive on the S3 backend with neither endpoint nor + # bucket; `atticExternal` names both and is the control that the + # assertion fires for the missing settings alone. + name = "the archive on S3 without an endpoint and a bucket fails evaluation, and with both it does not"; + ok = + let + refused = + m: + lib.any ( + a: !a.assertion && lib.hasInfix "deploy.archive.backend = \"s3\" requires" a.message + ) m.assertions; + in + refused halfCache && !(refused atticExternal); + } ]; in runGroup "builder" cases diff --git a/nix/vm-tests/swarm-archive.nix b/nix/vm-tests/swarm-archive.nix index 90858b699..aacc367f0 100644 --- a/nix/vm-tests/swarm-archive.nix +++ b/nix/vm-tests/swarm-archive.nix @@ -1,7 +1,9 @@ # `vmTests..swarm-archive`: runs the archive server's real units # (../host-modules/swarm-archive.nix) and the hive's real backup script # (../host-modules/lib/archive-backup.nix) in a VM, with restic and rclone, and -# checks what the module-eval suite can only read: +# checks what the module-eval suite can only read, on two nodes: `machine` keeps +# the repositories in a directory, `s3archive` in a bucket of a fake S3 +# (`rclone serve s3`). On both: # # - a backup of two agents lands in one repository each, and `work/` is left out; # - one agent's login cannot read another's repository; @@ -9,14 +11,18 @@ # applies the retention, also when a client has forged a snapshot dated in # the future; # - the users file follows the store: a user removed from it is refused without -# a restart of the server; -# - a hive that has agents and no grant backs up nothing and succeeds. +# a restart of the server. # -# One node, because the gateway hop is outside what is under test: the -# backup script dials `http://127.0.0.1:` instead of -# `https://archive.`, and the swarm secret store is a stand-in `bao` -# that serves `kv list`, `kv get` and `login` from a directory. The module-eval -# suite covers the gateway vhost and the store grants. +# Only on `machine`: a hive that has agents and no grant backs up nothing and +# succeeds. Only on `s3archive`: the server does not start while the store +# holds no S3 key; the repositories are objects in the bucket and not on the +# host; the key is in rclone's environment and in no unit or file. +# +# The gateway hop is outside what is under test: the backup script dials +# `http://127.0.0.1:` instead of `https://archive.`, and the swarm +# secret store is a stand-in `bao` that serves `kv list`, `kv get` and `login` +# from a directory. The module-eval suite covers the gateway vhost and the store +# grants. { pkgs }: let port = 8911; @@ -135,16 +141,26 @@ let excludes = [ ]; caBundle = null; }; -in -pkgs.testers.runNixOSTest { - name = "swarm-archive"; - nodes.machine = + # The external S3 of the second node: `rclone serve s3` over a directory, + # whose sub-directories are the buckets. The archive reaches it over HTTP + # with a key the unit generates at start, as it would reach a provider. rclone + # refuses short keys. + s3Port = 9000; + s3Root = "/var/lib/fake-s3"; + s3Bucket = "archive-bucket"; + s3KeyDir = "/var/lib/fake-s3-key"; + + # What both nodes share: the archive module, the stubs for the hive options it + # reads, and the hive's backup unit. + archiveNode = + extra: { pkgs, ... }: { imports = [ hiveStubs ../host-modules/swarm-archive.nix + extra ]; # The newest snapshot, plus whatever is younger than three days. @@ -174,23 +190,56 @@ pkgs.testers.runNixOSTest { script = backupScript; }; }; +in +pkgs.testers.runNixOSTest { + name = "swarm-archive"; + + nodes.machine = archiveNode { }; + + nodes.s3archive = archiveNode { + services.hyperhive.deploy.archive = { + backend = "s3"; + s3 = { + endpoint = "http://127.0.0.1:${toString s3Port}"; + bucket = s3Bucket; + prefix = "pfx"; + region = "us-east-1"; + }; + }; + + systemd.services.fake-s3 = { + wantedBy = [ "multi-user.target" ]; + before = [ "swarm-archive.service" ]; + environment.RCLONE_CONFIG = "/dev/null"; + script = '' + mkdir -p ${s3Root}/${s3Bucket} + install -d -m 0700 ${s3KeyDir} + if [ ! -e ${s3KeyDir}/id ]; then + head -c 15 /dev/urandom | base32 | tr -d = > ${s3KeyDir}/id + head -c 24 /dev/urandom | base32 | tr -d = > ${s3KeyDir}/secret + fi + exec ${pkgs.rclone}/bin/rclone serve s3 --addr 127.0.0.1:${toString s3Port} \ + --auth-key "$(cat ${s3KeyDir}/id),$(cat ${s3KeyDir}/secret)" ${s3Root} + ''; + }; + }; testScript = '' store = "${store}" url = "http://127.0.0.1:${toString port}" agents = ["alpha", "beta"] - def sha_line(agent, password): - return machine.succeed( + def sha_line(m, agent, password): + return m.succeed( f"printf '%s:{{SHA}}%s' {agent} " f"\"$(printf %s {password} | openssl dgst -sha1 -binary | base64)\"" ) - def put(path, field, value): - machine.succeed(f"mkdir -p {store}/{path} && printf %s '{value}' > {store}/{path}/{field}") + def put(m, path, field, value): + m.succeed(f"mkdir -p {store}/{path} && printf %s '{value}' > {store}/{path}/{field}") - def status(agent, repo, path="config", method="GET"): - return machine.succeed( + def status(m, agent, repo, path="config", method="GET"): + return m.succeed( f"curl -s -o /dev/null -w '%{{http_code}}' -X {method} " f"-u {agent}:restpw-{agent} {url}/{repo}/{path}" ) @@ -201,88 +250,99 @@ pkgs.testers.runNixOSTest { f"-r rest:http://{agent}:restpw-{agent}@127.0.0.1:${toString port}/{repo}/ {args}" ) + # What swarm-controller writes: a user line and a restic password per agent + # for the archive, a REST password and the restic password for the hive. + def grant_agents(m): + for agent in agents: + put(m, f"secret/swarm/services/archive/users/{agent}", "htpasswd", sha_line(m, agent, f"restpw-{agent}")) + put(m, f"secret/swarm/services/archive/restic/{agent}", "password", f"resticpw-{agent}") + put(m, f"secret/swarm/hives/${hive}/archive/{agent}", "rest-password", f"restpw-{agent}") + put(m, f"secret/swarm/hives/${hive}/archive/{agent}", "restic-password", f"resticpw-{agent}") + m.succeed( + f"mkdir -p ${agentRoot}/{agent}/state ${agentRoot}/{agent}/claude " + f"${agentRoot}/{agent}/harness ${agentRoot}/{agent}/work" + ) + m.succeed(f"echo {agent}-state > ${agentRoot}/{agent}/state/data") + m.succeed(f"echo {agent}-work > ${agentRoot}/{agent}/work/scratch") + + # What the archive promises, whichever backend holds the repositories. + def archive_suite(m, tag): + with subtest(f"{tag}: the server starts with no users and admits nobody"): + m.succeed("test -e /run/swarm-archive-auth/htpasswd") + assert status(m, "alpha", "alpha") == "401", status(m, "alpha", "alpha") + + grant_agents(m) + + with subtest(f"{tag}: the users file follows the store without a restart of the server"): + pid = m.succeed("systemctl show -p MainPID --value swarm-archive").strip() + m.succeed("systemctl start swarm-archive-users") + assert m.succeed("systemctl show -p MainPID --value swarm-archive").strip() == pid + assert status(m, "alpha", "alpha") in ("200", "404"), status(m, "alpha", "alpha") + assert status(m, "beta", "beta") in ("200", "404"), status(m, "beta", "beta") + # Wrong password: the control that the 200 above is a login, not an open door. + m.succeed(f"test $(curl -s -o /dev/null -w '%{{http_code}}' -u alpha:wrong {url}/alpha/config) = 401") + + with subtest(f"{tag}: the users file holds hashes, readable by the server only"): + m.succeed("grep -q '{SHA}' /run/swarm-archive-auth/htpasswd") + m.fail("grep -q restpw /run/swarm-archive-auth/htpasswd") + m.succeed("test \"$(stat -c '%a %U:%G' /run/swarm-archive-auth/htpasswd)\" = '640 root:swarm-archive'") + + with subtest(f"{tag}: the hive's backup puts each agent in its own repository, without work/"): + m.succeed("systemctl start test-archive-backup") + for agent in agents: + snaps = m.succeed(restic(agent, agent, "snapshots --json")) + assert snaps.count('"id"') == 1, snaps + listing = m.succeed(restic(agent, agent, "ls latest")) + assert f"/{agent}/state/data" in listing, listing + assert "scratch" not in listing, listing + + with subtest(f"{tag}: one agent's login cannot reach another's repository"): + assert status(m, "beta", "alpha") == "403", status(m, "beta", "alpha") + m.fail(restic("beta", "alpha", "snapshots")) + assert status(m, "alpha", "alpha") == "200", status(m, "alpha", "alpha") + assert status(m, "alpha", "beta") == "403", status(m, "alpha", "beta") + + with subtest(f"{tag}: a client cannot delete or overwrite"): + snap = m.succeed(restic("alpha", "alpha", "snapshots --json")).split('"id":"')[1].split('"')[0] + # An old snapshot for the prune to drop. The backup is also the presence control: a lock, which a backup creates and removes, is deletable. + m.succeed(restic("alpha", "alpha", "backup --host alpha --time '2000-01-01 00:00:00' ${agentRoot}/alpha/state")) + assert status(m, "alpha", "alpha", f"snapshots/{snap}", "DELETE") == "403" + assert status(m, "alpha", "alpha", "config", "POST") == "403" + m.fail(restic("alpha", "alpha", "forget --keep-last 1 --prune")) + assert m.succeed(restic("alpha", "alpha", "snapshots --json")).count('"id"') == 2 + + with subtest(f"{tag}: the server-side prune applies the retention"): + m.succeed("systemctl start swarm-archive-prune") + for agent in agents: + assert m.succeed(restic(agent, agent, "snapshots --json")).count('"id"') == 1 + # The old snapshot went, the one from the hive's backup stayed. + assert snap in m.succeed(restic("alpha", "alpha", "snapshots --json")) + + with subtest(f"{tag}: a snapshot dated in the future cannot push the real ones out of the retention"): + # restic counts --keep-within from the newest snapshot, and the client sets its time. + m.succeed(restic("alpha", "alpha", "backup --host alpha --time '2100-01-01 00:00:00' ${agentRoot}/alpha/state")) + assert m.succeed(restic("alpha", "alpha", "snapshots --json")).count('"id"') == 2 + m.succeed("systemctl start swarm-archive-prune") + left = m.succeed(restic("alpha", "alpha", "snapshots --json")) + assert left.count('"id"') == 1 and snap in left, left + m.succeed("journalctl -u swarm-archive-prune | grep -q 'archive prune warning: alpha has snapshots dated after now'") + # beta's repository had nothing forged, and says nothing. + m.fail("journalctl -u swarm-archive-prune | grep -q 'archive prune warning: beta'") + + with subtest(f"{tag}: a user removed from the store is refused without a restart"): + m.succeed(f"rm -r {store}/secret/swarm/services/archive/users/beta") + m.succeed("systemctl start swarm-archive-users") + assert status(m, "beta", "beta") == "401", status(m, "beta", "beta") + assert status(m, "alpha", "alpha") == "200", status(m, "alpha", "alpha") + + start_all() machine.wait_for_unit("multi-user.target") machine.wait_for_open_port(${toString port}) - with subtest("the server starts with no users and admits nobody"): - machine.succeed("test -e /run/swarm-archive-auth/htpasswd") - assert status("alpha", "alpha") == "401", status("alpha", "alpha") + archive_suite(machine, "local") - # What swarm-controller writes: a user line and a restic password per agent - # for the archive, a REST password and the restic password for the hive. - for agent in agents: - put(f"secret/swarm/services/archive/users/{agent}", "htpasswd", sha_line(agent, f"restpw-{agent}")) - put(f"secret/swarm/services/archive/restic/{agent}", "password", f"resticpw-{agent}") - put(f"secret/swarm/hives/${hive}/archive/{agent}", "rest-password", f"restpw-{agent}") - put(f"secret/swarm/hives/${hive}/archive/{agent}", "restic-password", f"resticpw-{agent}") - machine.succeed( - f"mkdir -p ${agentRoot}/{agent}/state ${agentRoot}/{agent}/claude " - f"${agentRoot}/{agent}/harness ${agentRoot}/{agent}/work" - ) - machine.succeed(f"echo {agent}-state > ${agentRoot}/{agent}/state/data") - machine.succeed(f"echo {agent}-work > ${agentRoot}/{agent}/work/scratch") - - with subtest("the users file follows the store without a restart of the server"): - pid = machine.succeed("systemctl show -p MainPID --value swarm-archive").strip() - machine.succeed("systemctl start swarm-archive-users") - assert machine.succeed("systemctl show -p MainPID --value swarm-archive").strip() == pid - assert status("alpha", "alpha") in ("200", "404"), status("alpha", "alpha") - assert status("beta", "beta") in ("200", "404"), status("beta", "beta") - # Wrong password: the control that the 200 above is a login, not an open door. - machine.succeed(f"test $(curl -s -o /dev/null -w '%{{http_code}}' -u alpha:wrong {url}/alpha/config) = 401") - - with subtest("the users file holds hashes, readable by the server only"): - machine.succeed("grep -q '{SHA}' /run/swarm-archive-auth/htpasswd") - machine.fail("grep -q restpw /run/swarm-archive-auth/htpasswd") - machine.succeed("test \"$(stat -c '%a %U:%G' /run/swarm-archive-auth/htpasswd)\" = '640 root:swarm-archive'") - - with subtest("the hive's backup puts each agent in its own repository, without work/"): - machine.succeed("systemctl start test-archive-backup") - for agent in agents: - snaps = machine.succeed(restic(agent, agent, "snapshots --json")) - assert snaps.count('"id"') == 1, snaps - listing = machine.succeed(restic(agent, agent, "ls latest")) - assert f"/{agent}/state/data" in listing, listing - assert "scratch" not in listing, listing - - with subtest("one agent's login cannot reach another's repository"): - assert status("beta", "alpha") == "403", status("beta", "alpha") - machine.fail(restic("beta", "alpha", "snapshots")) - assert status("alpha", "alpha") == "200", status("alpha", "alpha") - assert status("alpha", "beta") == "403", status("alpha", "beta") - - with subtest("a client cannot delete or overwrite"): - snap = machine.succeed(restic("alpha", "alpha", "snapshots --json")).split('"id":"')[1].split('"')[0] - # An old snapshot for the prune to drop. The backup is also the presence control: a lock, which a backup creates and removes, is deletable. - machine.succeed(restic("alpha", "alpha", "backup --host alpha --time '2000-01-01 00:00:00' ${agentRoot}/alpha/state")) - assert status("alpha", "alpha", f"snapshots/{snap}", "DELETE") == "403" - assert status("alpha", "alpha", "config", "POST") == "403" - machine.fail(restic("alpha", "alpha", "forget --keep-last 1 --prune")) - assert machine.succeed(restic("alpha", "alpha", "snapshots --json")).count('"id"') == 2 - - with subtest("the server-side prune applies the retention"): - machine.succeed("systemctl start swarm-archive-prune") - for agent in agents: - assert machine.succeed(restic(agent, agent, "snapshots --json")).count('"id"') == 1 - # The old snapshot went, the one from the hive's backup stayed. - assert snap in machine.succeed(restic("alpha", "alpha", "snapshots --json")) - - with subtest("a snapshot dated in the future cannot push the real ones out of the retention"): - # restic counts --keep-within from the newest snapshot, and the client sets its time. - machine.succeed(restic("alpha", "alpha", "backup --host alpha --time '2100-01-01 00:00:00' ${agentRoot}/alpha/state")) - assert machine.succeed(restic("alpha", "alpha", "snapshots --json")).count('"id"') == 2 - machine.succeed("systemctl start swarm-archive-prune") - left = machine.succeed(restic("alpha", "alpha", "snapshots --json")) - assert left.count('"id"') == 1 and snap in left, left - machine.succeed("journalctl -u swarm-archive-prune | grep -q 'archive prune warning: alpha has snapshots dated after now'") - # beta's repository had nothing forged, and says nothing. - machine.fail("journalctl -u swarm-archive-prune | grep -q 'archive prune warning: beta'") - - with subtest("a user removed from the store is refused without a restart"): - machine.succeed(f"rm -r {store}/secret/swarm/services/archive/users/beta") - machine.succeed("systemctl start swarm-archive-users") - assert status("beta", "beta") == "401", status("beta", "beta") - assert status("alpha", "alpha") == "200", status("alpha", "alpha") + with subtest("the repositories are under the data directory"): + machine.succeed("test -d /var/lib/swarm-archive/repos/alpha") with subtest("the archive announces itself once, so swarm-controller starts granting"): machine.succeed(f"test \"$(cat {store}/secret/swarm/services/archive/ready/ready)\" = 1") @@ -293,5 +353,45 @@ pkgs.testers.runNixOSTest { machine.succeed("systemctl start test-archive-backup") machine.succeed("journalctl -u test-archive-backup | grep -q 'granted this hive no archive credentials'") machine.fail("journalctl -u test-archive-backup | grep -q 'archive backup failed'") + + # The same archive on the second node, with its repositories on the fake S3. + s3archive.wait_for_unit("fake-s3.service") + s3_key_id = s3archive.succeed("cat ${s3KeyDir}/id").strip() + s3_secret_key = s3archive.succeed("cat ${s3KeyDir}/secret").strip() + + with subtest("s3: without the key in the store the server does not start"): + s3archive.wait_until_succeeds("journalctl -u swarm-archive | grep -q 'No value found'") + s3archive.fail("systemctl is-active --quiet swarm-archive") + s3archive.fail(f"curl -sf --max-time 2 -o /dev/null {url}/") + + with subtest("s3: the server starts once the store holds the key"): + put(s3archive, "secret/swarm/services/archive/s3/access-key-id", "value", s3_key_id) + put(s3archive, "secret/swarm/services/archive/s3/secret-access-key", "value", s3_secret_key) + s3archive.wait_for_open_port(${toString port}) + + archive_suite(s3archive, "s3") + + with subtest("s3: the repositories are in the bucket and not on this host"): + for agent in agents: + s3archive.succeed(f"test -f ${s3Root}/${s3Bucket}/pfx/{agent}/config") + s3archive.succeed("find ${s3Root}/${s3Bucket}/pfx/alpha/snapshots -type f | grep -q .") + s3archive.succeed("find ${s3Root}/${s3Bucket}/pfx/alpha/data -type f | grep -q .") + s3archive.fail("test -e /var/lib/swarm-archive/repos") + + with subtest("s3: the key is in rclone's environment and in no unit"): + pid = s3archive.succeed("systemctl show -p MainPID --value swarm-archive").strip() + env = s3archive.succeed(f"tr '\\0' '\\n' < /proc/{pid}/environ") + assert f"RCLONE_S3_ACCESS_KEY_ID={s3_key_id}" in env, env + assert f"RCLONE_S3_SECRET_ACCESS_KEY={s3_secret_key}" in env, env + # The connection settings are in the unit text, the control that the search + # for the key below looks at the units. + units = "swarm-archive swarm-archive-prune swarm-archive-users" + s3archive.succeed(f"systemctl cat {units} | grep -q RCLONE_S3_ENDPOINT") + s3archive.fail(f"systemctl cat {units} | grep -q {s3_key_id}") + s3archive.fail(f"systemctl cat {units} | grep -q {s3_secret_key}") + wrapper = s3archive.succeed("systemctl show -p ExecStart --value swarm-archive | grep -o '/nix/store/[^ ;]*swarm-archive-serve' | head -1").strip() + s3archive.succeed(f"grep -q 'kv get' {wrapper}") + s3archive.fail(f"grep -q {s3_key_id} {wrapper}") + s3archive.fail(f"grep -rq {s3_key_id} /run/swarm-archive-auth /var/cache/swarm-archive-prune") ''; }