swarm-archive: grant only once an archive is up; keep logins across a gap in placements
The archive host creates a ready marker in the store and swarm-controller writes no credentials before it exists, so a swarm without an archive, and a mixed-version swarm, leave every hive backup a logged no-op. A hive with no declaration no longer counts as placing nothing: its copies and its agents' user lines stay. The restic password is pinned by tests across a revoke, a re-placement and a move.
This commit is contained in:
atlas
parent
306aae0378
commit
91f36f39d2
12 changed files with 241 additions and 52 deletions
|
|
@ -43,6 +43,9 @@ Two properties protect the backups from a compromised hive:
|
|||
other. A hive holds the logins of the agents it places and no others. When an
|
||||
agent moves, the controller gives the new hive a new password and deletes the
|
||||
old hive's copy, so the old hive can't reach the repository any more.
|
||||
Only that login rotates. The controller creates an agent's restic password
|
||||
once, when it first places the agent, and a move, a destroy or a re-placement never
|
||||
changes it, because it's the key of the repository.
|
||||
- **Append-only.** A client can add snapshots and nothing else: no delete, no
|
||||
overwrite. The archive host removes old snapshots.
|
||||
|
||||
|
|
@ -98,23 +101,27 @@ users file refresh and the last prune.
|
|||
agent, and the lines above it say why. The alert waits 30 minutes, so an
|
||||
agent placed a few minutes ago doesn't trip it while its login is on its way.
|
||||
The next hourly run retries.
|
||||
- **`swarm-controller has granted this hive no archive credentials`.** The
|
||||
hive has agents and the store holds no login for any of them. Either
|
||||
`swarm-controller` doesn't have the archive yet (update it), or its archive
|
||||
pass fails: look for `archive:` lines in its journal.
|
||||
- **`swarm-controller has granted this hive no archive credentials`.** Not a
|
||||
failure: the hive logs it and succeeds. It means the store holds no login for
|
||||
any agent here, because no archive is running yet, `swarm-controller` is
|
||||
older than the archive, or its archive pass fails (look for `archive:` lines
|
||||
in its journal). If an archive is running and this persists, check the
|
||||
controller.
|
||||
- **The `archive prune failed` alert.** The daily prune couldn't apply the
|
||||
retention to one repository, for example because a backup held its lock for
|
||||
longer than ten minutes. The next prune retries.
|
||||
- **A swarm without an archive.** If no host sets `deploy.archive.enable`, every
|
||||
hive's run fails. Set `deploy.hive-controller.archive.enable = false` on the
|
||||
hives until one does.
|
||||
- **A swarm without an archive.** Nothing fails and nothing alerts. The archive
|
||||
host announces itself in the secret store when it starts, and
|
||||
`swarm-controller` writes no logins before that, so every hive's run finds
|
||||
nothing to back up and says so in its journal.
|
||||
|
||||
## Mixed versions
|
||||
|
||||
A hive that doesn't have the backup unit yet doesn't back up, and nothing else
|
||||
changes. A `swarm-controller` without the archive pass writes no logins, so
|
||||
hives with the unit fail with the message above until it updates. The archive
|
||||
host admits only the users the controller wrote, so until then it admits nobody.
|
||||
hives with the unit log that they have no credentials and succeed until it
|
||||
updates. The archive host admits only the users the controller wrote, so until
|
||||
then it admits nobody.
|
||||
|
||||
## What lives where
|
||||
|
||||
|
|
|
|||
|
|
@ -60,6 +60,7 @@ the rest of the cell says how.
|
|||
| `swarm/hives/<hive>/archive/<agent>` (fields `rest-password`, `restic-password`) | `swarm-controller`, for every agent a hive places, every 5 minutes: a new random REST password when the agent is new on that hive, the agent's restic password from the leaf below; the copy on a hive that no longer places the agent is deleted | the agent's hive: `hive-archive-backup`, under the hive's own store identity | ✅ a move to another hive writes a new REST password, which locks the old hive out | ✅ read at every backup run |
|
||||
| `swarm/services/archive/users/<agent>` (field `htpasswd`) | `swarm-controller`, with the REST password above: its `{SHA}` hash, never the password; deleted when no hive places the agent | `swarm-archive-users` on the archive's host, which renders the users file into `/run` | ✅ rewritten with the password | ✅ re-read every 2 minutes; the server notices the new file without a restart |
|
||||
| `swarm/services/archive/restic/<agent>` (field `password`) | `swarm-controller`, once per agent, on its first placement; never replaced and never deleted, because it opens every backup of the agent | `swarm-archive-prune` on the archive's host, through `RESTIC_PASSWORD_COMMAND` | ❌ a new password would not open the existing repository | ✅ read at every prune |
|
||||
| `swarm/services/archive/ready` | `swarm-archive-users` on the archive's host, once, when none is stored; `swarm-controller` writes no archive credentials before it exists | `swarm-controller`, every 5 minutes | ❌ never rewritten | ✅ read every 5 minutes |
|
||||
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
|
||||
|
||||
|
|
|
|||
|
|
@ -12,9 +12,9 @@
|
|||
# (../../swarm-controller/src/archive_access.rs), read with the hive's own
|
||||
# store identity: the agent's login to the archive, and its restic password,
|
||||
# which restic reads through `RESTIC_PASSWORD_COMMAND` and nothing writes to
|
||||
# disk. An agent the controller has not granted this hive is skipped; a hive
|
||||
# with agents and no grant at all fails, which is what a swarm-controller
|
||||
# without the archive looks like.
|
||||
# disk. An agent the controller has not granted this hive is skipped. The
|
||||
# controller grants nothing until a host runs the archive, so a swarm without
|
||||
# one, and a controller without this feature, leave every run a quiet no-op.
|
||||
#
|
||||
# A failed agent prints `archive backup failed: <agent>` to the journal,
|
||||
# which the swarm's `archive backup failed` alert reads, and fails the unit
|
||||
|
|
@ -65,9 +65,9 @@ in
|
|||
default = true;
|
||||
description = ''
|
||||
Back this hive's agents up to the swarm archive. On wherever the hive
|
||||
has its store identity; a swarm that runs no archive
|
||||
({option}`services.hyperhive.deploy.archive.enable` on one host) sees
|
||||
every run fail, so turn it off there.
|
||||
has its store identity. Where no host sets
|
||||
{option}`services.hyperhive.deploy.archive.enable`, every run finds no
|
||||
credentials, logs that and succeeds.
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -5,7 +5,8 @@
|
|||
# same script against a stand-in `bao`.
|
||||
#
|
||||
# Needs `bao`, `jq`, `coreutils`, `gnugrep`, `btrfs-progs` on the unit's `path`
|
||||
# and the `BAO_*` client set. Every failure prints a line starting
|
||||
# and the `BAO_*` client set. A store that cannot be reached, or an agent whose
|
||||
# backup fails, prints a line starting
|
||||
# `archive backup failed`, which the swarm's alert reads, and ends the run with
|
||||
# a non-zero status once the other agents have been tried.
|
||||
{ lib }:
|
||||
|
|
@ -127,11 +128,10 @@ in
|
|||
fi
|
||||
done
|
||||
|
||||
# Agents here and not one of them granted: swarm-controller has not written
|
||||
# this hive's grants, which a controller without the archive never does.
|
||||
# No grant for any agent here: the swarm has no archive, or its controller
|
||||
# has not granted this hive yet. Neither is a failure of this hive.
|
||||
if [ "$candidates" -gt 0 ] && [ -z "$granted" ]; then
|
||||
echo "archive backup failed: swarm-controller has granted this hive no archive credentials ($candidates agents here)" >&2
|
||||
failed=1
|
||||
echo "swarm-controller has granted this hive no archive credentials ($candidates agents here); nothing to back up."
|
||||
fi
|
||||
exit "$failed"
|
||||
''
|
||||
|
|
|
|||
|
|
@ -31,6 +31,7 @@ let
|
|||
|
||||
usersPrefix = "secret/swarm/services/archive/users";
|
||||
resticPrefix = "secret/swarm/services/archive/restic";
|
||||
readyPath = "secret/swarm/services/archive/ready";
|
||||
|
||||
keepFlags = lib.concatStringsSep " " (
|
||||
[ "--keep-last ${toString retention.keepLast}" ]
|
||||
|
|
@ -81,6 +82,12 @@ in
|
|||
exit 1
|
||||
fi
|
||||
|
||||
# Tells swarm-controller an archive exists; it grants no credentials until
|
||||
# then. Created once, never rewritten.
|
||||
if ! store_bao kv get -field=ready ${lib.escapeShellArg readyPath} >/dev/null 2>&1; then
|
||||
store_bao kv put -cas=0 ${lib.escapeShellArg readyPath} ready=1 >/dev/null
|
||||
fi
|
||||
|
||||
# A prefix that holds nothing is "No value found"; it means no agent has a
|
||||
# user yet, not a failure.
|
||||
rc=0
|
||||
|
|
|
|||
|
|
@ -16,6 +16,8 @@
|
|||
# - `swarm-archive-users` (root, every two minutes) renders the htpasswd file
|
||||
# from the users leaves into `/run`, which is tmpfs. rclone re-reads the file
|
||||
# when it changes. The leaves hold password hashes only.
|
||||
# The unit also creates `swarm/services/archive/ready`, which tells
|
||||
# swarm-controller an archive exists: it grants no credentials before that.
|
||||
# - `swarm-archive-prune` (daily) reads each agent's restic password through
|
||||
# `RESTIC_PASSWORD_COMMAND` for its `restic forget --prune`. This host can
|
||||
# therefore decrypt every backup, which is the cost of pruning where clients
|
||||
|
|
|
|||
|
|
@ -546,6 +546,10 @@ let
|
|||
capabilities = ["list"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/data/swarm/services/archive/ready" {
|
||||
capabilities = ["read"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/metadata/swarm/services/archive/users/+" {
|
||||
capabilities = ["delete"]
|
||||
}
|
||||
|
|
@ -1139,6 +1143,12 @@ let
|
|||
}
|
||||
'';
|
||||
|
||||
createReadStanza = path: ''
|
||||
path "${path}" {
|
||||
capabilities = ["create", "read"]
|
||||
}
|
||||
'';
|
||||
|
||||
listStanza = path: ''
|
||||
path "${path}" {
|
||||
capabilities = ["list"]
|
||||
|
|
@ -1383,7 +1393,8 @@ let
|
|||
|
||||
# The backup archive: the users file is rendered from one leaf per agent, and
|
||||
# the prune reads each agent's restic password. Read and list only; the
|
||||
# controller writes both.
|
||||
# controller writes both. It also creates the marker that tells the
|
||||
# controller an archive exists.
|
||||
archiveReaders = [
|
||||
{
|
||||
name = "swarm-archive";
|
||||
|
|
@ -1391,7 +1402,8 @@ let
|
|||
policyText =
|
||||
readStanza "${credentialMountPath}/data/swarm/services/archive/users/*"
|
||||
+ readStanza "${credentialMountPath}/data/swarm/services/archive/restic/*"
|
||||
+ listStanza "${credentialMountPath}/metadata/swarm/services/archive/users/";
|
||||
+ listStanza "${credentialMountPath}/metadata/swarm/services/archive/users/"
|
||||
+ createReadStanza "${credentialMountPath}/data/swarm/services/archive/ready";
|
||||
}
|
||||
];
|
||||
|
||||
|
|
|
|||
|
|
@ -2171,7 +2171,7 @@ let
|
|||
# The archive reads the users and the repositories' passwords and writes
|
||||
# nothing; the hive-side copies of the REST password are not its to read.
|
||||
# The controller is the only writer: the control is its own grant.
|
||||
name = "the archive's grant is read on its users and restic passwords, list on its users, and nothing else, and only the controller writes them";
|
||||
name = "the archive's grant is read on its users and restic passwords, list on its users, create and read on its ready marker, and nothing else, and only the controller writes them";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-archive-policy.script;
|
||||
|
|
@ -2181,8 +2181,14 @@ let
|
|||
(grantFor grants "secret/data/swarm/services/archive/users/atlas").caps == [ "read" ]
|
||||
&& (grantFor grants "secret/data/swarm/services/archive/restic/atlas").caps == [ "read" ]
|
||||
&& (grantFor grants "secret/metadata/swarm/services/archive/users/").caps == [ "list" ]
|
||||
&&
|
||||
(grantFor grants "secret/data/swarm/services/archive/ready").caps == [
|
||||
"create"
|
||||
"read"
|
||||
]
|
||||
&& (grantFor c "secret/data/swarm/services/archive/ready").caps == [ "read" ]
|
||||
&& grantFor grants "secret/data/swarm/hives/h1/archive/atlas" == null
|
||||
&& lib.length grants == 3
|
||||
&& lib.length grants == 4
|
||||
&& lib.hasInfix "allowed_common_names=swarm-archive" s
|
||||
&& lib.hasInfix "token_policies=swarm-archive" s
|
||||
&& lib.elem "create" (grantFor c "secret/data/swarm/services/archive/users/atlas").caps
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
# applies the retention;
|
||||
# - the users file follows the store: a user removed from it is refused without
|
||||
# a restart of the server;
|
||||
# - a hive that has agents and no grant fails its backup.
|
||||
# - a hive that has agents and no grant backs up nothing and succeeds.
|
||||
#
|
||||
# One node, because the gateway hop is outside what is under test: the
|
||||
# backup script dials `http://127.0.0.1:<port>` instead of
|
||||
|
|
@ -33,6 +33,11 @@ let
|
|||
[ -d "$store/$path" ] || { echo "No value found at $path" >&2; exit 2; }
|
||||
ls "$store/$path" | ${pkgs.jq}/bin/jq -R . | ${pkgs.jq}/bin/jq -s .
|
||||
exit 0 ;;
|
||||
"kv put")
|
||||
[ -e "$store/$4" ] && { echo "check-and-set parameter did not match" >&2; exit 2; }
|
||||
mkdir -p "$store/$4"
|
||||
printf %s "''${5#*=}" > "$store/$4/''${5%%=*}"
|
||||
exit 0 ;;
|
||||
"kv get")
|
||||
field="''${3#-field=}"
|
||||
[ -f "$store/$4/$field" ] || { echo "No value found at $4" >&2; exit 2; }
|
||||
|
|
@ -265,9 +270,14 @@ pkgs.testers.runNixOSTest {
|
|||
assert status("beta", "beta") == "401", status("beta", "beta")
|
||||
assert status("alpha", "alpha") == "200", status("alpha", "alpha")
|
||||
|
||||
with subtest("a hive with agents and no grant fails its backup"):
|
||||
with subtest("the archive announces itself once, so swarm-controller starts granting"):
|
||||
machine.succeed(f"test \"$(cat {store}/secret/swarm/services/archive/ready/ready)\" = 1")
|
||||
machine.succeed("systemctl start swarm-archive-users")
|
||||
|
||||
with subtest("a hive with agents and no grant backs up nothing and does not fail"):
|
||||
machine.succeed(f"rm -r {store}/secret/swarm/hives/${hive}")
|
||||
machine.fail("systemctl start test-archive-backup")
|
||||
machine.succeed("journalctl -u test-archive-backup | grep -q 'archive backup failed: swarm-controller has granted this hive no archive credentials'")
|
||||
machine.succeed("systemctl start test-archive-backup")
|
||||
machine.succeed("journalctl -u test-archive-backup | grep -q 'granted this hive no archive credentials'")
|
||||
machine.fail("journalctl -u test-archive-backup | grep -q 'archive backup failed'")
|
||||
'';
|
||||
}
|
||||
|
|
|
|||
|
|
@ -199,21 +199,37 @@ async fn leaves(store: &impl ArchiveStore, prefix: &str) -> Result<Vec<String>>
|
|||
/// One pass: grant every agent in `placed`, then revoke what no placement
|
||||
/// justifies. `hives` is every hive of the swarm, whose copies are checked.
|
||||
///
|
||||
/// Nothing happens until the archive has announced itself at
|
||||
/// [`archive::ready_path`]: a swarm without one gets no credentials. A hive in
|
||||
/// `undeclared` has no declaration to read, which does not show it places
|
||||
/// nothing, so its copies, and the user lines of the agents they name, stay.
|
||||
///
|
||||
/// # Errors
|
||||
/// Never: a failure is counted in [`Done::failed`] and logged, and the rest of
|
||||
/// the pass still runs.
|
||||
pub(crate) async fn reconcile(
|
||||
store: &impl ArchiveStore,
|
||||
placed: &BTreeSet<(String, String)>,
|
||||
undeclared: &BTreeSet<String>,
|
||||
hives: &[String],
|
||||
) -> Done {
|
||||
let mut done = Done::default();
|
||||
match store.read(&archive::ready_path().unwrap_or_default()).await {
|
||||
Ok(Some(_)) => {}
|
||||
Ok(None) => return done,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %format!("{e:#}"), "archive: reading whether the archive is up failed");
|
||||
done.failed += 1;
|
||||
return done;
|
||||
}
|
||||
}
|
||||
let mut holders: BTreeMap<&str, Vec<&str>> = BTreeMap::new();
|
||||
for (hive, agent) in placed {
|
||||
holders.entry(agent).or_default().push(hive);
|
||||
}
|
||||
|
||||
let mut ambiguous = BTreeSet::new();
|
||||
let mut unconfirmed = BTreeSet::new();
|
||||
for (agent, on) in &holders {
|
||||
if let [hive] = on.as_slice() {
|
||||
match grant(store, hive, agent).await {
|
||||
|
|
@ -243,6 +259,10 @@ pub(crate) async fn reconcile(
|
|||
continue;
|
||||
}
|
||||
};
|
||||
if undeclared.contains(hive) {
|
||||
unconfirmed.extend(copies);
|
||||
continue;
|
||||
}
|
||||
for agent in copies {
|
||||
let keep = ambiguous.contains(agent.as_str())
|
||||
|| placed.contains(&(hive.clone(), agent.clone()));
|
||||
|
|
@ -255,7 +275,7 @@ pub(crate) async fn reconcile(
|
|||
match leaves(store, &archive::users_prefix().unwrap_or_default()).await {
|
||||
Ok(users) => {
|
||||
for agent in users {
|
||||
if !holders.contains_key(agent.as_str()) {
|
||||
if !holders.contains_key(agent.as_str()) && !unconfirmed.contains(&agent) {
|
||||
revoke(store, archive::user_path(&agent), &mut done).await;
|
||||
}
|
||||
}
|
||||
|
|
@ -303,11 +323,12 @@ pub fn spawn(wanted: Arc<WantedWriter>, hives: Vec<String>) {
|
|||
loop {
|
||||
ticker.tick().await;
|
||||
let pass = async {
|
||||
let placed = crate::jobq::placements(&wanted, &hives).await?;
|
||||
let (placed, undeclared) =
|
||||
crate::jobq::placements_with_gaps(&wanted, &hives).await?;
|
||||
let store = crate::store::connect()
|
||||
.await
|
||||
.context("logging in to the swarm secret store")?;
|
||||
anyhow::Ok(reconcile(&store, &placed, &hives).await)
|
||||
anyhow::Ok(reconcile(&store, &placed, &undeclared, &hives).await)
|
||||
};
|
||||
match tokio::time::timeout(PASS_DEADLINE, pass).await {
|
||||
Ok(Ok(done)) if done != Done::default() => tracing::info!(
|
||||
|
|
@ -347,6 +368,16 @@ mod tests {
|
|||
}
|
||||
|
||||
impl Memory {
|
||||
/// A store whose archive has announced itself.
|
||||
fn up() -> Self {
|
||||
let store = Self::default();
|
||||
store.kv.lock().unwrap().insert(
|
||||
archive::ready_path().unwrap(),
|
||||
fields(&[(archive::READY, "1")]),
|
||||
);
|
||||
store
|
||||
}
|
||||
|
||||
fn get(&self, path: &str) -> Option<Fields> {
|
||||
self.kv.lock().unwrap().get(path).cloned()
|
||||
}
|
||||
|
|
@ -414,6 +445,10 @@ mod tests {
|
|||
.collect()
|
||||
}
|
||||
|
||||
fn none() -> BTreeSet<String> {
|
||||
BTreeSet::new()
|
||||
}
|
||||
|
||||
fn hives() -> Vec<String> {
|
||||
vec!["alpha".to_owned(), "beta".to_owned()]
|
||||
}
|
||||
|
|
@ -434,8 +469,8 @@ mod tests {
|
|||
|
||||
#[tokio::test]
|
||||
async fn a_placed_agent_gets_its_hive_a_password_and_the_archive_its_hash() {
|
||||
let store = Memory::default();
|
||||
let done = reconcile(&store, &placed(&[("alpha", "atlas")]), &hives()).await;
|
||||
let store = Memory::up();
|
||||
let done = reconcile(&store, &placed(&[("alpha", "atlas")]), &none(), &hives()).await;
|
||||
assert_eq!(
|
||||
done,
|
||||
Done {
|
||||
|
|
@ -460,23 +495,23 @@ mod tests {
|
|||
|
||||
#[tokio::test]
|
||||
async fn a_current_grant_is_not_rewritten() {
|
||||
let store = Memory::default();
|
||||
let store = Memory::up();
|
||||
let one = placed(&[("alpha", "atlas")]);
|
||||
reconcile(&store, &one, &hives()).await;
|
||||
reconcile(&store, &one, &none(), &hives()).await;
|
||||
let before = store.get(ALPHA_ATLAS);
|
||||
let done = reconcile(&store, &one, &hives()).await;
|
||||
let done = reconcile(&store, &one, &none(), &hives()).await;
|
||||
assert_eq!(done, Done::default());
|
||||
assert_eq!(store.get(ALPHA_ATLAS), before);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_move_replaces_the_password_and_takes_the_old_hives_copy() {
|
||||
let store = Memory::default();
|
||||
reconcile(&store, &placed(&[("alpha", "atlas")]), &hives()).await;
|
||||
let store = Memory::up();
|
||||
reconcile(&store, &placed(&[("alpha", "atlas")]), &none(), &hives()).await;
|
||||
let old_rest = store.value(ALPHA_ATLAS, REST_PASSWORD).unwrap();
|
||||
let restic = store.value(RESTIC_ATLAS, PASSWORD).unwrap();
|
||||
|
||||
let done = reconcile(&store, &placed(&[("beta", "atlas")]), &hives()).await;
|
||||
let done = reconcile(&store, &placed(&[("beta", "atlas")]), &none(), &hives()).await;
|
||||
assert_eq!(done.granted, 1);
|
||||
assert_eq!(done.revoked, 1);
|
||||
assert!(store.get(ALPHA_ATLAS).is_none(), "the old hive loses read");
|
||||
|
|
@ -499,14 +534,15 @@ mod tests {
|
|||
|
||||
#[tokio::test]
|
||||
async fn a_destroyed_agent_loses_every_credential_but_the_restic_password() {
|
||||
let store = Memory::default();
|
||||
let store = Memory::up();
|
||||
reconcile(
|
||||
&store,
|
||||
&placed(&[("alpha", "atlas"), ("beta", "argus")]),
|
||||
&none(),
|
||||
&hives(),
|
||||
)
|
||||
.await;
|
||||
let done = reconcile(&store, &placed(&[("beta", "argus")]), &hives()).await;
|
||||
let done = reconcile(&store, &placed(&[("beta", "argus")]), &none(), &hives()).await;
|
||||
assert_eq!(done.revoked, 2);
|
||||
assert!(store.get(ALPHA_ATLAS).is_none());
|
||||
assert!(store.get(USER_ATLAS).is_none());
|
||||
|
|
@ -521,30 +557,34 @@ mod tests {
|
|||
|
||||
#[tokio::test]
|
||||
async fn an_agent_two_hives_place_is_left_as_it_is() {
|
||||
let store = Memory::default();
|
||||
reconcile(&store, &placed(&[("alpha", "atlas")]), &hives()).await;
|
||||
let store = Memory::up();
|
||||
reconcile(&store, &placed(&[("alpha", "atlas")]), &none(), &hives()).await;
|
||||
let before = (store.get(ALPHA_ATLAS), store.get(USER_ATLAS));
|
||||
let both = placed(&[("alpha", "atlas"), ("beta", "atlas")]);
|
||||
let done = reconcile(&store, &both, &hives()).await;
|
||||
let done = reconcile(&store, &both, &none(), &hives()).await;
|
||||
assert_eq!(done, Done::default());
|
||||
assert_eq!((store.get(ALPHA_ATLAS), store.get(USER_ATLAS)), before);
|
||||
assert!(store.get(BETA_ATLAS).is_none());
|
||||
|
||||
// Neither hive is granted when the agent is new and both place it.
|
||||
let fresh = Memory::default();
|
||||
let done = reconcile(&fresh, &both, &hives()).await;
|
||||
let fresh = Memory::up();
|
||||
let done = reconcile(&fresh, &both, &none(), &hives()).await;
|
||||
assert_eq!(done, Done::default());
|
||||
assert!(fresh.kv.lock().unwrap().is_empty());
|
||||
assert_eq!(
|
||||
fresh.kv.lock().unwrap().keys().collect::<Vec<_>>(),
|
||||
[&archive::ready_path().unwrap()],
|
||||
"only the archive's own marker is stored"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_lost_user_line_is_rewritten_from_the_hives_password() {
|
||||
let store = Memory::default();
|
||||
let store = Memory::up();
|
||||
let one = placed(&[("alpha", "atlas")]);
|
||||
reconcile(&store, &one, &hives()).await;
|
||||
reconcile(&store, &one, &none(), &hives()).await;
|
||||
let line = store.value(USER_ATLAS, HTPASSWD);
|
||||
store.kv.lock().unwrap().remove(USER_ATLAS);
|
||||
let done = reconcile(&store, &one, &hives()).await;
|
||||
let done = reconcile(&store, &one, &none(), &hives()).await;
|
||||
assert_eq!(done.granted, 1);
|
||||
assert_eq!(store.value(USER_ATLAS, HTPASSWD), line);
|
||||
}
|
||||
|
|
@ -553,11 +593,12 @@ mod tests {
|
|||
async fn a_refused_write_counts_as_failed_and_the_other_agents_still_get_theirs() {
|
||||
let store = Memory {
|
||||
refuse: Some("swarm/hives/alpha/"),
|
||||
..Memory::default()
|
||||
..Memory::up()
|
||||
};
|
||||
let done = reconcile(
|
||||
&store,
|
||||
&placed(&[("alpha", "atlas"), ("beta", "argus")]),
|
||||
&none(),
|
||||
&hives(),
|
||||
)
|
||||
.await;
|
||||
|
|
@ -569,4 +610,73 @@ mod tests {
|
|||
"no hash without the hive's copy"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn the_restic_password_survives_a_revoke_and_a_new_grant_and_a_move() {
|
||||
let store = Memory::up();
|
||||
let atlas = |hive: &str| placed(&[(hive, "atlas")]);
|
||||
reconcile(&store, &atlas("alpha"), &none(), &hives()).await;
|
||||
let restic = store.value(RESTIC_ATLAS, PASSWORD).expect("minted once");
|
||||
let rest = store.value(ALPHA_ATLAS, REST_PASSWORD).expect("granted");
|
||||
|
||||
// Nothing places the agent: every credential goes, the restic one stays.
|
||||
let nowhere = placed(&[]);
|
||||
reconcile(&store, &nowhere, &none(), &hives()).await;
|
||||
assert!(store.get(ALPHA_ATLAS).is_none() && store.get(USER_ATLAS).is_none());
|
||||
assert_eq!(store.value(RESTIC_ATLAS, PASSWORD).as_ref(), Some(&restic));
|
||||
|
||||
// Placed again: the login is new, the repository's password is the one it was created with.
|
||||
reconcile(&store, &atlas("alpha"), &none(), &hives()).await;
|
||||
assert_ne!(store.value(ALPHA_ATLAS, REST_PASSWORD), Some(rest));
|
||||
assert_eq!(
|
||||
store.value(ALPHA_ATLAS, RESTIC_PASSWORD).as_ref(),
|
||||
Some(&restic)
|
||||
);
|
||||
assert_eq!(store.value(RESTIC_ATLAS, PASSWORD).as_ref(), Some(&restic));
|
||||
|
||||
// And across a move.
|
||||
reconcile(&store, &atlas("beta"), &none(), &hives()).await;
|
||||
assert_eq!(
|
||||
store.value(BETA_ATLAS, RESTIC_PASSWORD).as_ref(),
|
||||
Some(&restic)
|
||||
);
|
||||
assert_eq!(store.value(RESTIC_ATLAS, PASSWORD).as_ref(), Some(&restic));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_hive_with_no_declaration_keeps_its_copies_and_its_agents_user_lines() {
|
||||
let store = Memory::up();
|
||||
reconcile(&store, &placed(&[("alpha", "atlas")]), &none(), &hives()).await;
|
||||
let (copy, line) = (store.get(ALPHA_ATLAS), store.get(USER_ATLAS));
|
||||
|
||||
// alpha's declaration is not there: it does not place atlas, and that proves nothing.
|
||||
let gap: BTreeSet<String> = ["alpha".to_owned()].into();
|
||||
let done = reconcile(&store, &placed(&[]), &gap, &hives()).await;
|
||||
assert_eq!(done, Done::default());
|
||||
assert_eq!(
|
||||
(store.get(ALPHA_ATLAS), store.get(USER_ATLAS)),
|
||||
(copy, line)
|
||||
);
|
||||
|
||||
// Control: with the declaration read, the same placements revoke.
|
||||
let done = reconcile(&store, &placed(&[]), &none(), &hives()).await;
|
||||
assert_eq!(done.revoked, 2);
|
||||
assert!(store.get(ALPHA_ATLAS).is_none() && store.get(USER_ATLAS).is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_swarm_whose_archive_has_not_announced_itself_gets_nothing() {
|
||||
let store = Memory::default();
|
||||
let done = reconcile(&store, &placed(&[("alpha", "atlas")]), &none(), &hives()).await;
|
||||
assert_eq!(done, Done::default());
|
||||
assert!(store.kv.lock().unwrap().is_empty());
|
||||
|
||||
// Control: the same placements, once the archive is up.
|
||||
store.kv.lock().unwrap().insert(
|
||||
archive::ready_path().unwrap(),
|
||||
fields(&[(archive::READY, "1")]),
|
||||
);
|
||||
let done = reconcile(&store, &placed(&[("alpha", "atlas")]), &none(), &hives()).await;
|
||||
assert_eq!(done.granted, 1);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -965,13 +965,28 @@ pub(crate) async fn placements(
|
|||
wanted: &wanted::WantedWriter,
|
||||
hives: &[String],
|
||||
) -> Result<std::collections::BTreeSet<(String, String)>> {
|
||||
Ok(placements_with_gaps(wanted, hives).await?.0)
|
||||
}
|
||||
|
||||
/// [`placements`], and the hives that have no declaration at all. Such a hive
|
||||
/// places nothing here, but the queue cannot tell a hive that never had agents
|
||||
/// from one whose declaration is not there (yet).
|
||||
pub(crate) async fn placements_with_gaps(
|
||||
wanted: &wanted::WantedWriter,
|
||||
hives: &[String],
|
||||
) -> Result<(
|
||||
std::collections::BTreeSet<(String, String)>,
|
||||
std::collections::BTreeSet<String>,
|
||||
)> {
|
||||
let mut placed = std::collections::BTreeSet::new();
|
||||
let mut undeclared = std::collections::BTreeSet::new();
|
||||
for hive in hives {
|
||||
let Some(declared) = wanted
|
||||
.view(hive)
|
||||
.await
|
||||
.map_err(|e| e.context(format!("read where hive {hive} places agents")))?
|
||||
else {
|
||||
undeclared.insert(hive.clone());
|
||||
continue;
|
||||
};
|
||||
placed.extend(
|
||||
|
|
@ -982,7 +997,7 @@ pub(crate) async fn placements(
|
|||
.map(|(name, _)| (hive.clone(), name)),
|
||||
);
|
||||
}
|
||||
Ok(placed)
|
||||
Ok((placed, undeclared))
|
||||
}
|
||||
|
||||
/// Every agent in `placements`, with its archetype.
|
||||
|
|
|
|||
|
|
@ -35,6 +35,21 @@ pub const PASSWORD: &str = "password";
|
|||
/// The field of [`user_path`] holding the htpasswd line.
|
||||
pub const HTPASSWD: &str = "htpasswd";
|
||||
|
||||
/// The field of [`ready_path`].
|
||||
pub const READY: &str = "ready";
|
||||
|
||||
/// Where the archive host announces itself. `swarm-controller` writes no
|
||||
/// credentials until this exists, so a swarm without an archive gets none.
|
||||
///
|
||||
/// # Errors
|
||||
/// Never in practice: the segments are constants.
|
||||
pub fn ready_path() -> Result<String, Error> {
|
||||
Ok(format!(
|
||||
"{}/ready",
|
||||
principal_prefix(Kind::Service, SERVICE)?
|
||||
))
|
||||
}
|
||||
|
||||
/// The directory holding every [`user_path`], to list.
|
||||
///
|
||||
/// # Errors
|
||||
|
|
@ -104,6 +119,10 @@ mod tests {
|
|||
grant_path("alpha", "atlas").expect("plain names"),
|
||||
"swarm/hives/alpha/archive/atlas"
|
||||
);
|
||||
assert_eq!(
|
||||
ready_path().expect("constant"),
|
||||
"swarm/services/archive/ready"
|
||||
);
|
||||
assert_eq!(
|
||||
users_prefix().expect("constant"),
|
||||
"swarm/services/archive/users"
|
||||
|
|
|
|||
Loading…
Reference in a new issue