From 91f36f39d2d498ff8495cb647250ff7b857e66fa Mon Sep 17 00:00:00 2001 From: atlas Date: Tue, 10 Nov 2026 08:18:45 +0100 Subject: [PATCH] swarm-archive: grant only once an archive is up; keep logins across a gap in placements The archive host creates a ready marker in the store and swarm-controller writes no credentials before it exists, so a swarm without an archive, and a mixed-version swarm, leave every hive backup a logged no-op. A hive with no declaration no longer counts as placing nothing: its copies and its agents' user lines stay. The restic password is pinned by tests across a revoke, a re-placement and a move. --- docs/2-handbook/swarm/archive.md | 25 ++- .../2-handbook/swarm/credentials-reference.md | 1 + nix/host-modules/hive-archive-backup.nix | 12 +- nix/host-modules/lib/archive-backup.nix | 10 +- nix/host-modules/lib/archive-server.nix | 7 + nix/host-modules/swarm-archive.nix | 2 + nix/host-modules/swarm-bao.nix | 16 +- nix/module-eval/bao-grants.nix | 10 +- nix/vm-tests/swarm-archive.nix | 18 +- swarm-controller/src/archive_access.rs | 156 +++++++++++++++--- swarm-controller/src/jobq.rs | 17 +- swarm-secret-client/src/archive.rs | 19 +++ 12 files changed, 241 insertions(+), 52 deletions(-) diff --git a/docs/2-handbook/swarm/archive.md b/docs/2-handbook/swarm/archive.md index c448f1e15..b31c28c49 100644 --- a/docs/2-handbook/swarm/archive.md +++ b/docs/2-handbook/swarm/archive.md @@ -43,6 +43,9 @@ Two properties protect the backups from a compromised hive: other. A hive holds the logins of the agents it places and no others. When an agent moves, the controller gives the new hive a new password and deletes the old hive's copy, so the old hive can't reach the repository any more. + Only that login rotates. The controller creates an agent's restic password + once, when it first places the agent, and a move, a destroy or a re-placement never + changes it, because it's the key of the repository. - **Append-only.** A client can add snapshots and nothing else: no delete, no overwrite. The archive host removes old snapshots. @@ -98,23 +101,27 @@ users file refresh and the last prune. agent, and the lines above it say why. The alert waits 30 minutes, so an agent placed a few minutes ago doesn't trip it while its login is on its way. The next hourly run retries. -- **`swarm-controller has granted this hive no archive credentials`.** The - hive has agents and the store holds no login for any of them. Either - `swarm-controller` doesn't have the archive yet (update it), or its archive - pass fails: look for `archive:` lines in its journal. +- **`swarm-controller has granted this hive no archive credentials`.** Not a + failure: the hive logs it and succeeds. It means the store holds no login for + any agent here, because no archive is running yet, `swarm-controller` is + older than the archive, or its archive pass fails (look for `archive:` lines + in its journal). If an archive is running and this persists, check the + controller. - **The `archive prune failed` alert.** The daily prune couldn't apply the retention to one repository, for example because a backup held its lock for longer than ten minutes. The next prune retries. -- **A swarm without an archive.** If no host sets `deploy.archive.enable`, every - hive's run fails. Set `deploy.hive-controller.archive.enable = false` on the - hives until one does. +- **A swarm without an archive.** Nothing fails and nothing alerts. The archive + host announces itself in the secret store when it starts, and + `swarm-controller` writes no logins before that, so every hive's run finds + nothing to back up and says so in its journal. ## Mixed versions A hive that doesn't have the backup unit yet doesn't back up, and nothing else changes. A `swarm-controller` without the archive pass writes no logins, so -hives with the unit fail with the message above until it updates. The archive -host admits only the users the controller wrote, so until then it admits nobody. +hives with the unit log that they have no credentials and succeed until it +updates. The archive host admits only the users the controller wrote, so until +then it admits nobody. ## What lives where diff --git a/docs/2-handbook/swarm/credentials-reference.md b/docs/2-handbook/swarm/credentials-reference.md index 125044a2a..ec55d7b81 100644 --- a/docs/2-handbook/swarm/credentials-reference.md +++ b/docs/2-handbook/swarm/credentials-reference.md @@ -60,6 +60,7 @@ the rest of the cell says how. | `swarm/hives//archive/` (fields `rest-password`, `restic-password`) | `swarm-controller`, for every agent a hive places, every 5 minutes: a new random REST password when the agent is new on that hive, the agent's restic password from the leaf below; the copy on a hive that no longer places the agent is deleted | the agent's hive: `hive-archive-backup`, under the hive's own store identity | ✅ a move to another hive writes a new REST password, which locks the old hive out | ✅ read at every backup run | | `swarm/services/archive/users/` (field `htpasswd`) | `swarm-controller`, with the REST password above: its `{SHA}` hash, never the password; deleted when no hive places the agent | `swarm-archive-users` on the archive's host, which renders the users file into `/run` | ✅ rewritten with the password | ✅ re-read every 2 minutes; the server notices the new file without a restart | | `swarm/services/archive/restic/` (field `password`) | `swarm-controller`, once per agent, on its first placement; never replaced and never deleted, because it opens every backup of the agent | `swarm-archive-prune` on the archive's host, through `RESTIC_PASSWORD_COMMAND` | ❌ a new password would not open the existing repository | ✅ read at every prune | +| `swarm/services/archive/ready` | `swarm-archive-users` on the archive's host, once, when none is stored; `swarm-controller` writes no archive credentials before it exists | `swarm-controller`, every 5 minutes | ❌ never rewritten | ✅ read every 5 minutes | | `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass | | _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated | diff --git a/nix/host-modules/hive-archive-backup.nix b/nix/host-modules/hive-archive-backup.nix index 240068fbf..ae47985ac 100644 --- a/nix/host-modules/hive-archive-backup.nix +++ b/nix/host-modules/hive-archive-backup.nix @@ -12,9 +12,9 @@ # (../../swarm-controller/src/archive_access.rs), read with the hive's own # store identity: the agent's login to the archive, and its restic password, # which restic reads through `RESTIC_PASSWORD_COMMAND` and nothing writes to -# disk. An agent the controller has not granted this hive is skipped; a hive -# with agents and no grant at all fails, which is what a swarm-controller -# without the archive looks like. +# disk. An agent the controller has not granted this hive is skipped. The +# controller grants nothing until a host runs the archive, so a swarm without +# one, and a controller without this feature, leave every run a quiet no-op. # # A failed agent prints `archive backup failed: ` to the journal, # which the swarm's `archive backup failed` alert reads, and fails the unit @@ -65,9 +65,9 @@ in default = true; description = '' Back this hive's agents up to the swarm archive. On wherever the hive - has its store identity; a swarm that runs no archive - ({option}`services.hyperhive.deploy.archive.enable` on one host) sees - every run fail, so turn it off there. + has its store identity. Where no host sets + {option}`services.hyperhive.deploy.archive.enable`, every run finds no + credentials, logs that and succeeds. ''; }; diff --git a/nix/host-modules/lib/archive-backup.nix b/nix/host-modules/lib/archive-backup.nix index 22009dd5a..8af4f0f29 100644 --- a/nix/host-modules/lib/archive-backup.nix +++ b/nix/host-modules/lib/archive-backup.nix @@ -5,7 +5,8 @@ # same script against a stand-in `bao`. # # Needs `bao`, `jq`, `coreutils`, `gnugrep`, `btrfs-progs` on the unit's `path` -# and the `BAO_*` client set. Every failure prints a line starting +# and the `BAO_*` client set. A store that cannot be reached, or an agent whose +# backup fails, prints a line starting # `archive backup failed`, which the swarm's alert reads, and ends the run with # a non-zero status once the other agents have been tried. { lib }: @@ -127,11 +128,10 @@ in fi done - # Agents here and not one of them granted: swarm-controller has not written - # this hive's grants, which a controller without the archive never does. + # No grant for any agent here: the swarm has no archive, or its controller + # has not granted this hive yet. Neither is a failure of this hive. if [ "$candidates" -gt 0 ] && [ -z "$granted" ]; then - echo "archive backup failed: swarm-controller has granted this hive no archive credentials ($candidates agents here)" >&2 - failed=1 + echo "swarm-controller has granted this hive no archive credentials ($candidates agents here); nothing to back up." fi exit "$failed" '' diff --git a/nix/host-modules/lib/archive-server.nix b/nix/host-modules/lib/archive-server.nix index 368693c11..c76082242 100644 --- a/nix/host-modules/lib/archive-server.nix +++ b/nix/host-modules/lib/archive-server.nix @@ -31,6 +31,7 @@ let usersPrefix = "secret/swarm/services/archive/users"; resticPrefix = "secret/swarm/services/archive/restic"; + readyPath = "secret/swarm/services/archive/ready"; keepFlags = lib.concatStringsSep " " ( [ "--keep-last ${toString retention.keepLast}" ] @@ -81,6 +82,12 @@ in exit 1 fi + # Tells swarm-controller an archive exists; it grants no credentials until + # then. Created once, never rewritten. + if ! store_bao kv get -field=ready ${lib.escapeShellArg readyPath} >/dev/null 2>&1; then + store_bao kv put -cas=0 ${lib.escapeShellArg readyPath} ready=1 >/dev/null + fi + # A prefix that holds nothing is "No value found"; it means no agent has a # user yet, not a failure. rc=0 diff --git a/nix/host-modules/swarm-archive.nix b/nix/host-modules/swarm-archive.nix index 777669a59..fb896fe1e 100644 --- a/nix/host-modules/swarm-archive.nix +++ b/nix/host-modules/swarm-archive.nix @@ -16,6 +16,8 @@ # - `swarm-archive-users` (root, every two minutes) renders the htpasswd file # from the users leaves into `/run`, which is tmpfs. rclone re-reads the file # when it changes. The leaves hold password hashes only. +# The unit also creates `swarm/services/archive/ready`, which tells +# swarm-controller an archive exists: it grants no credentials before that. # - `swarm-archive-prune` (daily) reads each agent's restic password through # `RESTIC_PASSWORD_COMMAND` for its `restic forget --prune`. This host can # therefore decrypt every backup, which is the cost of pruning where clients diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index ce1e54d78..bd01dda99 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -546,6 +546,10 @@ let capabilities = ["list"] } + path "${credentialMountPath}/data/swarm/services/archive/ready" { + capabilities = ["read"] + } + path "${credentialMountPath}/metadata/swarm/services/archive/users/+" { capabilities = ["delete"] } @@ -1139,6 +1143,12 @@ let } ''; + createReadStanza = path: '' + path "${path}" { + capabilities = ["create", "read"] + } + ''; + listStanza = path: '' path "${path}" { capabilities = ["list"] @@ -1383,7 +1393,8 @@ let # The backup archive: the users file is rendered from one leaf per agent, and # the prune reads each agent's restic password. Read and list only; the - # controller writes both. + # controller writes both. It also creates the marker that tells the + # controller an archive exists. archiveReaders = [ { name = "swarm-archive"; @@ -1391,7 +1402,8 @@ let policyText = readStanza "${credentialMountPath}/data/swarm/services/archive/users/*" + readStanza "${credentialMountPath}/data/swarm/services/archive/restic/*" - + listStanza "${credentialMountPath}/metadata/swarm/services/archive/users/"; + + listStanza "${credentialMountPath}/metadata/swarm/services/archive/users/" + + createReadStanza "${credentialMountPath}/data/swarm/services/archive/ready"; } ]; diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index 4223dc26e..c8395f348 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -2171,7 +2171,7 @@ let # The archive reads the users and the repositories' passwords and writes # nothing; the hive-side copies of the REST password are not its to read. # The controller is the only writer: the control is its own grant. - name = "the archive's grant is read on its users and restic passwords, list on its users, and nothing else, and only the controller writes them"; + name = "the archive's grant is read on its users and restic passwords, list on its users, create and read on its ready marker, and nothing else, and only the controller writes them"; ok = let s = baoGrantHere.systemd.services.swarm-bao-archive-policy.script; @@ -2181,8 +2181,14 @@ let (grantFor grants "secret/data/swarm/services/archive/users/atlas").caps == [ "read" ] && (grantFor grants "secret/data/swarm/services/archive/restic/atlas").caps == [ "read" ] && (grantFor grants "secret/metadata/swarm/services/archive/users/").caps == [ "list" ] + && + (grantFor grants "secret/data/swarm/services/archive/ready").caps == [ + "create" + "read" + ] + && (grantFor c "secret/data/swarm/services/archive/ready").caps == [ "read" ] && grantFor grants "secret/data/swarm/hives/h1/archive/atlas" == null - && lib.length grants == 3 + && lib.length grants == 4 && lib.hasInfix "allowed_common_names=swarm-archive" s && lib.hasInfix "token_policies=swarm-archive" s && lib.elem "create" (grantFor c "secret/data/swarm/services/archive/users/atlas").caps diff --git a/nix/vm-tests/swarm-archive.nix b/nix/vm-tests/swarm-archive.nix index c40d4f649..90c6ae77b 100644 --- a/nix/vm-tests/swarm-archive.nix +++ b/nix/vm-tests/swarm-archive.nix @@ -9,7 +9,7 @@ # applies the retention; # - the users file follows the store: a user removed from it is refused without # a restart of the server; -# - a hive that has agents and no grant fails its backup. +# - a hive that has agents and no grant backs up nothing and succeeds. # # One node, because the gateway hop is outside what is under test: the # backup script dials `http://127.0.0.1:` instead of @@ -33,6 +33,11 @@ let [ -d "$store/$path" ] || { echo "No value found at $path" >&2; exit 2; } ls "$store/$path" | ${pkgs.jq}/bin/jq -R . | ${pkgs.jq}/bin/jq -s . exit 0 ;; + "kv put") + [ -e "$store/$4" ] && { echo "check-and-set parameter did not match" >&2; exit 2; } + mkdir -p "$store/$4" + printf %s "''${5#*=}" > "$store/$4/''${5%%=*}" + exit 0 ;; "kv get") field="''${3#-field=}" [ -f "$store/$4/$field" ] || { echo "No value found at $4" >&2; exit 2; } @@ -265,9 +270,14 @@ pkgs.testers.runNixOSTest { assert status("beta", "beta") == "401", status("beta", "beta") assert status("alpha", "alpha") == "200", status("alpha", "alpha") - with subtest("a hive with agents and no grant fails its backup"): + with subtest("the archive announces itself once, so swarm-controller starts granting"): + machine.succeed(f"test \"$(cat {store}/secret/swarm/services/archive/ready/ready)\" = 1") + machine.succeed("systemctl start swarm-archive-users") + + with subtest("a hive with agents and no grant backs up nothing and does not fail"): machine.succeed(f"rm -r {store}/secret/swarm/hives/${hive}") - machine.fail("systemctl start test-archive-backup") - machine.succeed("journalctl -u test-archive-backup | grep -q 'archive backup failed: swarm-controller has granted this hive no archive credentials'") + machine.succeed("systemctl start test-archive-backup") + machine.succeed("journalctl -u test-archive-backup | grep -q 'granted this hive no archive credentials'") + machine.fail("journalctl -u test-archive-backup | grep -q 'archive backup failed'") ''; } diff --git a/swarm-controller/src/archive_access.rs b/swarm-controller/src/archive_access.rs index 0df949fb5..8da2e3b9e 100644 --- a/swarm-controller/src/archive_access.rs +++ b/swarm-controller/src/archive_access.rs @@ -199,21 +199,37 @@ async fn leaves(store: &impl ArchiveStore, prefix: &str) -> Result> /// One pass: grant every agent in `placed`, then revoke what no placement /// justifies. `hives` is every hive of the swarm, whose copies are checked. /// +/// Nothing happens until the archive has announced itself at +/// [`archive::ready_path`]: a swarm without one gets no credentials. A hive in +/// `undeclared` has no declaration to read, which does not show it places +/// nothing, so its copies, and the user lines of the agents they name, stay. +/// /// # Errors /// Never: a failure is counted in [`Done::failed`] and logged, and the rest of /// the pass still runs. pub(crate) async fn reconcile( store: &impl ArchiveStore, placed: &BTreeSet<(String, String)>, + undeclared: &BTreeSet, hives: &[String], ) -> Done { let mut done = Done::default(); + match store.read(&archive::ready_path().unwrap_or_default()).await { + Ok(Some(_)) => {} + Ok(None) => return done, + Err(e) => { + tracing::warn!(error = %format!("{e:#}"), "archive: reading whether the archive is up failed"); + done.failed += 1; + return done; + } + } let mut holders: BTreeMap<&str, Vec<&str>> = BTreeMap::new(); for (hive, agent) in placed { holders.entry(agent).or_default().push(hive); } let mut ambiguous = BTreeSet::new(); + let mut unconfirmed = BTreeSet::new(); for (agent, on) in &holders { if let [hive] = on.as_slice() { match grant(store, hive, agent).await { @@ -243,6 +259,10 @@ pub(crate) async fn reconcile( continue; } }; + if undeclared.contains(hive) { + unconfirmed.extend(copies); + continue; + } for agent in copies { let keep = ambiguous.contains(agent.as_str()) || placed.contains(&(hive.clone(), agent.clone())); @@ -255,7 +275,7 @@ pub(crate) async fn reconcile( match leaves(store, &archive::users_prefix().unwrap_or_default()).await { Ok(users) => { for agent in users { - if !holders.contains_key(agent.as_str()) { + if !holders.contains_key(agent.as_str()) && !unconfirmed.contains(&agent) { revoke(store, archive::user_path(&agent), &mut done).await; } } @@ -303,11 +323,12 @@ pub fn spawn(wanted: Arc, hives: Vec) { loop { ticker.tick().await; let pass = async { - let placed = crate::jobq::placements(&wanted, &hives).await?; + let (placed, undeclared) = + crate::jobq::placements_with_gaps(&wanted, &hives).await?; let store = crate::store::connect() .await .context("logging in to the swarm secret store")?; - anyhow::Ok(reconcile(&store, &placed, &hives).await) + anyhow::Ok(reconcile(&store, &placed, &undeclared, &hives).await) }; match tokio::time::timeout(PASS_DEADLINE, pass).await { Ok(Ok(done)) if done != Done::default() => tracing::info!( @@ -347,6 +368,16 @@ mod tests { } impl Memory { + /// A store whose archive has announced itself. + fn up() -> Self { + let store = Self::default(); + store.kv.lock().unwrap().insert( + archive::ready_path().unwrap(), + fields(&[(archive::READY, "1")]), + ); + store + } + fn get(&self, path: &str) -> Option { self.kv.lock().unwrap().get(path).cloned() } @@ -414,6 +445,10 @@ mod tests { .collect() } + fn none() -> BTreeSet { + BTreeSet::new() + } + fn hives() -> Vec { vec!["alpha".to_owned(), "beta".to_owned()] } @@ -434,8 +469,8 @@ mod tests { #[tokio::test] async fn a_placed_agent_gets_its_hive_a_password_and_the_archive_its_hash() { - let store = Memory::default(); - let done = reconcile(&store, &placed(&[("alpha", "atlas")]), &hives()).await; + let store = Memory::up(); + let done = reconcile(&store, &placed(&[("alpha", "atlas")]), &none(), &hives()).await; assert_eq!( done, Done { @@ -460,23 +495,23 @@ mod tests { #[tokio::test] async fn a_current_grant_is_not_rewritten() { - let store = Memory::default(); + let store = Memory::up(); let one = placed(&[("alpha", "atlas")]); - reconcile(&store, &one, &hives()).await; + reconcile(&store, &one, &none(), &hives()).await; let before = store.get(ALPHA_ATLAS); - let done = reconcile(&store, &one, &hives()).await; + let done = reconcile(&store, &one, &none(), &hives()).await; assert_eq!(done, Done::default()); assert_eq!(store.get(ALPHA_ATLAS), before); } #[tokio::test] async fn a_move_replaces_the_password_and_takes_the_old_hives_copy() { - let store = Memory::default(); - reconcile(&store, &placed(&[("alpha", "atlas")]), &hives()).await; + let store = Memory::up(); + reconcile(&store, &placed(&[("alpha", "atlas")]), &none(), &hives()).await; let old_rest = store.value(ALPHA_ATLAS, REST_PASSWORD).unwrap(); let restic = store.value(RESTIC_ATLAS, PASSWORD).unwrap(); - let done = reconcile(&store, &placed(&[("beta", "atlas")]), &hives()).await; + let done = reconcile(&store, &placed(&[("beta", "atlas")]), &none(), &hives()).await; assert_eq!(done.granted, 1); assert_eq!(done.revoked, 1); assert!(store.get(ALPHA_ATLAS).is_none(), "the old hive loses read"); @@ -499,14 +534,15 @@ mod tests { #[tokio::test] async fn a_destroyed_agent_loses_every_credential_but_the_restic_password() { - let store = Memory::default(); + let store = Memory::up(); reconcile( &store, &placed(&[("alpha", "atlas"), ("beta", "argus")]), + &none(), &hives(), ) .await; - let done = reconcile(&store, &placed(&[("beta", "argus")]), &hives()).await; + let done = reconcile(&store, &placed(&[("beta", "argus")]), &none(), &hives()).await; assert_eq!(done.revoked, 2); assert!(store.get(ALPHA_ATLAS).is_none()); assert!(store.get(USER_ATLAS).is_none()); @@ -521,30 +557,34 @@ mod tests { #[tokio::test] async fn an_agent_two_hives_place_is_left_as_it_is() { - let store = Memory::default(); - reconcile(&store, &placed(&[("alpha", "atlas")]), &hives()).await; + let store = Memory::up(); + reconcile(&store, &placed(&[("alpha", "atlas")]), &none(), &hives()).await; let before = (store.get(ALPHA_ATLAS), store.get(USER_ATLAS)); let both = placed(&[("alpha", "atlas"), ("beta", "atlas")]); - let done = reconcile(&store, &both, &hives()).await; + let done = reconcile(&store, &both, &none(), &hives()).await; assert_eq!(done, Done::default()); assert_eq!((store.get(ALPHA_ATLAS), store.get(USER_ATLAS)), before); assert!(store.get(BETA_ATLAS).is_none()); // Neither hive is granted when the agent is new and both place it. - let fresh = Memory::default(); - let done = reconcile(&fresh, &both, &hives()).await; + let fresh = Memory::up(); + let done = reconcile(&fresh, &both, &none(), &hives()).await; assert_eq!(done, Done::default()); - assert!(fresh.kv.lock().unwrap().is_empty()); + assert_eq!( + fresh.kv.lock().unwrap().keys().collect::>(), + [&archive::ready_path().unwrap()], + "only the archive's own marker is stored" + ); } #[tokio::test] async fn a_lost_user_line_is_rewritten_from_the_hives_password() { - let store = Memory::default(); + let store = Memory::up(); let one = placed(&[("alpha", "atlas")]); - reconcile(&store, &one, &hives()).await; + reconcile(&store, &one, &none(), &hives()).await; let line = store.value(USER_ATLAS, HTPASSWD); store.kv.lock().unwrap().remove(USER_ATLAS); - let done = reconcile(&store, &one, &hives()).await; + let done = reconcile(&store, &one, &none(), &hives()).await; assert_eq!(done.granted, 1); assert_eq!(store.value(USER_ATLAS, HTPASSWD), line); } @@ -553,11 +593,12 @@ mod tests { async fn a_refused_write_counts_as_failed_and_the_other_agents_still_get_theirs() { let store = Memory { refuse: Some("swarm/hives/alpha/"), - ..Memory::default() + ..Memory::up() }; let done = reconcile( &store, &placed(&[("alpha", "atlas"), ("beta", "argus")]), + &none(), &hives(), ) .await; @@ -569,4 +610,73 @@ mod tests { "no hash without the hive's copy" ); } + + #[tokio::test] + async fn the_restic_password_survives_a_revoke_and_a_new_grant_and_a_move() { + let store = Memory::up(); + let atlas = |hive: &str| placed(&[(hive, "atlas")]); + reconcile(&store, &atlas("alpha"), &none(), &hives()).await; + let restic = store.value(RESTIC_ATLAS, PASSWORD).expect("minted once"); + let rest = store.value(ALPHA_ATLAS, REST_PASSWORD).expect("granted"); + + // Nothing places the agent: every credential goes, the restic one stays. + let nowhere = placed(&[]); + reconcile(&store, &nowhere, &none(), &hives()).await; + assert!(store.get(ALPHA_ATLAS).is_none() && store.get(USER_ATLAS).is_none()); + assert_eq!(store.value(RESTIC_ATLAS, PASSWORD).as_ref(), Some(&restic)); + + // Placed again: the login is new, the repository's password is the one it was created with. + reconcile(&store, &atlas("alpha"), &none(), &hives()).await; + assert_ne!(store.value(ALPHA_ATLAS, REST_PASSWORD), Some(rest)); + assert_eq!( + store.value(ALPHA_ATLAS, RESTIC_PASSWORD).as_ref(), + Some(&restic) + ); + assert_eq!(store.value(RESTIC_ATLAS, PASSWORD).as_ref(), Some(&restic)); + + // And across a move. + reconcile(&store, &atlas("beta"), &none(), &hives()).await; + assert_eq!( + store.value(BETA_ATLAS, RESTIC_PASSWORD).as_ref(), + Some(&restic) + ); + assert_eq!(store.value(RESTIC_ATLAS, PASSWORD).as_ref(), Some(&restic)); + } + + #[tokio::test] + async fn a_hive_with_no_declaration_keeps_its_copies_and_its_agents_user_lines() { + let store = Memory::up(); + reconcile(&store, &placed(&[("alpha", "atlas")]), &none(), &hives()).await; + let (copy, line) = (store.get(ALPHA_ATLAS), store.get(USER_ATLAS)); + + // alpha's declaration is not there: it does not place atlas, and that proves nothing. + let gap: BTreeSet = ["alpha".to_owned()].into(); + let done = reconcile(&store, &placed(&[]), &gap, &hives()).await; + assert_eq!(done, Done::default()); + assert_eq!( + (store.get(ALPHA_ATLAS), store.get(USER_ATLAS)), + (copy, line) + ); + + // Control: with the declaration read, the same placements revoke. + let done = reconcile(&store, &placed(&[]), &none(), &hives()).await; + assert_eq!(done.revoked, 2); + assert!(store.get(ALPHA_ATLAS).is_none() && store.get(USER_ATLAS).is_none()); + } + + #[tokio::test] + async fn a_swarm_whose_archive_has_not_announced_itself_gets_nothing() { + let store = Memory::default(); + let done = reconcile(&store, &placed(&[("alpha", "atlas")]), &none(), &hives()).await; + assert_eq!(done, Done::default()); + assert!(store.kv.lock().unwrap().is_empty()); + + // Control: the same placements, once the archive is up. + store.kv.lock().unwrap().insert( + archive::ready_path().unwrap(), + fields(&[(archive::READY, "1")]), + ); + let done = reconcile(&store, &placed(&[("alpha", "atlas")]), &none(), &hives()).await; + assert_eq!(done.granted, 1); + } } diff --git a/swarm-controller/src/jobq.rs b/swarm-controller/src/jobq.rs index dcfd2a47b..0873a048d 100644 --- a/swarm-controller/src/jobq.rs +++ b/swarm-controller/src/jobq.rs @@ -965,13 +965,28 @@ pub(crate) async fn placements( wanted: &wanted::WantedWriter, hives: &[String], ) -> Result> { + Ok(placements_with_gaps(wanted, hives).await?.0) +} + +/// [`placements`], and the hives that have no declaration at all. Such a hive +/// places nothing here, but the queue cannot tell a hive that never had agents +/// from one whose declaration is not there (yet). +pub(crate) async fn placements_with_gaps( + wanted: &wanted::WantedWriter, + hives: &[String], +) -> Result<( + std::collections::BTreeSet<(String, String)>, + std::collections::BTreeSet, +)> { let mut placed = std::collections::BTreeSet::new(); + let mut undeclared = std::collections::BTreeSet::new(); for hive in hives { let Some(declared) = wanted .view(hive) .await .map_err(|e| e.context(format!("read where hive {hive} places agents")))? else { + undeclared.insert(hive.clone()); continue; }; placed.extend( @@ -982,7 +997,7 @@ pub(crate) async fn placements( .map(|(name, _)| (hive.clone(), name)), ); } - Ok(placed) + Ok((placed, undeclared)) } /// Every agent in `placements`, with its archetype. diff --git a/swarm-secret-client/src/archive.rs b/swarm-secret-client/src/archive.rs index 1bf3c18fb..0fac10a08 100644 --- a/swarm-secret-client/src/archive.rs +++ b/swarm-secret-client/src/archive.rs @@ -35,6 +35,21 @@ pub const PASSWORD: &str = "password"; /// The field of [`user_path`] holding the htpasswd line. pub const HTPASSWD: &str = "htpasswd"; +/// The field of [`ready_path`]. +pub const READY: &str = "ready"; + +/// Where the archive host announces itself. `swarm-controller` writes no +/// credentials until this exists, so a swarm without an archive gets none. +/// +/// # Errors +/// Never in practice: the segments are constants. +pub fn ready_path() -> Result { + Ok(format!( + "{}/ready", + principal_prefix(Kind::Service, SERVICE)? + )) +} + /// The directory holding every [`user_path`], to list. /// /// # Errors @@ -104,6 +119,10 @@ mod tests { grant_path("alpha", "atlas").expect("plain names"), "swarm/hives/alpha/archive/atlas" ); + assert_eq!( + ready_path().expect("constant"), + "swarm/services/archive/ready" + ); assert_eq!( users_prefix().expect("constant"), "swarm/services/archive/users"