feat: migrate from sqlite to VSS (WIP)

[rolznz]

On the hub side, I would update the Alby Account page to allow the user to manually enable VSS (currently a one-way upgrade), which would save user configs (LdkVssEnabled=true and LdkMigrateStorage="VSS") and shut down the node, and then ask them to login. When they login, the user config entry will be checked, removed, and the migration to VSS will occur, using builder.MigrateStorage(ldk_node.MigrateStorageVss). (in the same way we have methods to reset some node state right now)

How it works:

1. We rename the existing LDK-node sqlite DB as a backup file.
2. We create the VSS store, with a fresh secondary storage (a new LDK-node sqlite DB)
3. We migrate essential data from the backup file to VSS. (channel manager, channel monitors, and list of peers.)

There could be some edge cases to consider (e.g. if there are pending channel monitor updates we should not proceed with the migration), and some TODOs to address.

LDK-nodes original migration script is below, but not all the keys are needed, and I'm not sure it's 100% correct.

/// **Warning**: This will overwrite existing data in VSS for the node.
	pub fn migrate_data_from_fs_to_vss(
		&self, vss_url: String, store_id: String, header_provider: Arc<dyn VssHeaderProvider>,
	) -> Result<(), BuildError> {
		let logger = setup_logger(&self.config)?;

		let seed_bytes = seed_bytes_from_config(
			&self.config,
			self.entropy_source_config.as_ref(),
			Arc::clone(&logger),
		)?;

		let config = Arc::new(self.config.clone());

		let vss_xprv = derive_vss_xprv(config.clone(), &seed_bytes, Arc::clone(&logger))?;

		let vss_seed_bytes: [u8; 32] = vss_xprv.private_key.secret_bytes();

		let mut storage_dir_path: PathBuf = self.config.storage_dir_path.clone().into();
		storage_dir_path.push("fs_store");

		fs::create_dir_all(storage_dir_path.clone())
			.map_err(|_| BuildError::StoragePathAccessFailed)?;
		let fs_store = FilesystemStore::new(storage_dir_path);

		let vss_store =
			VssStore::new(vss_url, store_id, vss_seed_bytes, header_provider).map_err(|e| {
				log_error!(logger, "Failed to setup VssStore: {}", e);
				BuildError::KVStoreSetupFailed
			})?;

		let namespace_pairs = [
			(
				io::EVENT_QUEUE_PERSISTENCE_PRIMARY_NAMESPACE,
				io::EVENT_QUEUE_PERSISTENCE_SECONDARY_NAMESPACE,
			),
			(
				io::PEER_INFO_PERSISTENCE_PRIMARY_NAMESPACE,
				io::PEER_INFO_PERSISTENCE_SECONDARY_NAMESPACE,
			),
			(
				io::PAYMENT_INFO_PERSISTENCE_PRIMARY_NAMESPACE,
				io::PAYMENT_INFO_PERSISTENCE_SECONDARY_NAMESPACE,
			),
			(
				io::DEPRECATED_SPENDABLE_OUTPUT_INFO_PERSISTENCE_PRIMARY_NAMESPACE,
				io::DEPRECATED_SPENDABLE_OUTPUT_INFO_PERSISTENCE_SECONDARY_NAMESPACE,
			),
			(io::NODE_METRICS_PRIMARY_NAMESPACE, io::NODE_METRICS_SECONDARY_NAMESPACE),
			(
				io::BDK_WALLET_DESCRIPTOR_PRIMARY_NAMESPACE,
				io::BDK_WALLET_DESCRIPTOR_SECONDARY_NAMESPACE,
			),
			(
				io::BDK_WALLET_CHANGE_DESCRIPTOR_PRIMARY_NAMESPACE,
				io::BDK_WALLET_CHANGE_DESCRIPTOR_SECONDARY_NAMESPACE,
			),
			(io::BDK_WALLET_NETWORK_PRIMARY_NAMESPACE, io::BDK_WALLET_NETWORK_SECONDARY_NAMESPACE),
			(
				io::BDK_WALLET_LOCAL_CHAIN_PRIMARY_NAMESPACE,
				io::BDK_WALLET_LOCAL_CHAIN_SECONDARY_NAMESPACE,
			),
			(io::BDK_WALLET_TX_GRAPH_PRIMARY_NAMESPACE, io::BDK_WALLET_TX_GRAPH_PRIMARY_NAMESPACE),
			(io::BDK_WALLET_INDEXER_PRIMARY_NAMESPACE, io::BDK_WALLET_INDEXER_SECONDARY_NAMESPACE),
			(CHANNEL_MANAGER_PERSISTENCE_PRIMARY_NAMESPACE, CHANNEL_MANAGER_PERSISTENCE_SECONDARY_NAMESPACE),
			(CHANNEL_MONITOR_PERSISTENCE_PRIMARY_NAMESPACE, CHANNEL_MONITOR_PERSISTENCE_SECONDARY_NAMESPACE),
			(ARCHIVED_CHANNEL_MONITOR_PERSISTENCE_PRIMARY_NAMESPACE, ARCHIVED_CHANNEL_MONITOR_PERSISTENCE_SECONDARY_NAMESPACE),
			(NETWORK_GRAPH_PERSISTENCE_PRIMARY_NAMESPACE, NETWORK_GRAPH_PERSISTENCE_SECONDARY_NAMESPACE),
			(SCORER_PERSISTENCE_PRIMARY_NAMESPACE, SCORER_PERSISTENCE_SECONDARY_NAMESPACE),
			(OUTPUT_SWEEPER_PERSISTENCE_PRIMARY_NAMESPACE, OUTPUT_SWEEPER_PERSISTENCE_SECONDARY_NAMESPACE),
			(OUTPUT_SWEEPER_PERSISTENCE_PRIMARY_NAMESPACE, OUTPUT_SWEEPER_PERSISTENCE_SECONDARY_NAMESPACE),
		];

		Self::migrate_kvstore(
			&fs_store as &dyn KVStore,
			&vss_store as &dyn KVStore,
			&namespace_pairs,
		)
			.unwrap();

		Ok(())
	}

	fn migrate_kvstore(
		source: &dyn KVStore, dest: &dyn KVStore, namespace_pairs: &[(&str, &str)],
	) -> Result<(), std::io::Error> {
		let unique_namespace_pairs: std::collections::HashSet<(&str, &str)> =
			namespace_pairs.iter().cloned().collect();
		let unique_namespace_pairs: Vec<(&str, &str)> =
			unique_namespace_pairs.into_iter().collect();

		for &(primary_namespace, secondary_namespace) in &unique_namespace_pairs {
			let keys = source.list(primary_namespace, secondary_namespace)?;
			for key in keys {
				let value = source.read(primary_namespace, secondary_namespace, &key)?;
				dest.write(primary_namespace, secondary_namespace, &key, &value)?
			}
		}
		Ok(())
	}

[rdmitr]

Left a couple minor comments :)

Also, do we actually need to do this in the builder?.. Maybe we could expose a separate FFI function, implemented in a separate module even, that would copy data from one KVStore into another KVStore (in a generic manner ideally, so that we could automagically implement migrations in different directions)? And then proceed with normal node build with VSS process? That way, we will clearly separate the migration process, reducing divergence with the upstream, as well as make sure the migration runs and completes before LDK-node even starts doing its initialization. The separate migration module could even have a CLI frontend 🤔

[rolznz]

Left a couple minor comments :)

@rdmitr I can't see the comments (maybe you need to finish the review?)

Also, do we actually need to do this in the builder?.. Maybe we could expose a separate FFI function, implemented in a separate module even, that would copy data from one KVStore into another KVStore (in a generic manner ideally, so that we could automagically implement migrations in different directions)? And then proceed with normal node build with VSS process? That way, we will clearly separate the migration process, reducing divergence with the upstream, as well as make sure the migration runs and completes before LDK-node even starts doing its initialization. The separate migration module could even have a CLI frontend 🤔

I don't think moving this into a separate tool is good because there's more chance the user runs it while their alby hub is running, which risks corrupting the migration, and this increases friction when ideally we want users to migrate as soon as possible. If people have to go through multiple steps and download extra software to do this, I think things will go wrong and less people will do it. I think this also might be used more, as users will need to migrate once they update their subscription, or if they cancel their subscription.

Also, this is an imperfect migration. To properly migrate from VSS back to sqlite we would need to clear the all the VSS data, which I think is not possible right now (This needs a change upstream in rust-lightning to be able to list all keys across all namespaces to be able to clear them). I would prefer to wait for LDK to implement this and officially implement migrations before we support migrating back from VSS to Sqlite. For me the priority here is to enable a one way upgrade from Sqlite to VSS.

@bumi do you have any thoughts?

[bumi]

as discussed I think moving it in a separate module sounds good to me. (I always like moving things out :D )
optimizing for migrating back and forth is not needed. people should decide before hand. So it should be a one off thing.