feat(bulk-load): bulk load download part2 - replica download files

[hycdong]

This pull request add do_download function and its unit test. This function is used to download files from remote provider.

[acelyc111]

After redownloaded, better to check local file is exist and md5sum matched.

[hycdong]

I used block_file_mock function here, it is a mock function, whose download function is like below:

rdsn/src/dist/replication/test/replica_test/unit_test/block_service_mock.h

Lines 93 to 99 in f5ab808

	virtual dsn::task_ptr download(const download_request &req,
	dsn::task_code code,
	const download_callback &cb,
	dsn::task_tracker *tracker = nullptr)
	{
	return task_ptr();
	}

It will not do actual downloading action, the local file is not existed in unit test. And the real downloading action is hard to mock.

[acelyc111]

After downloaded, check local file is exist and md5sum matched.

[neverchanje]

I found nothing in your codes relates to the replica except for ddebug_replica. It should be moved to dist::block_service module, rather than be in replica.

struct download_options {
  std::string &remote_dir;
  std::string &local_dir;
  // TODO(): add download rate limiter here.
  // uint64_t max_download_size;
};

error_s download_remote_files(const download_options& options, dist::block_service::block_filesystem *fs, /*out*/ uint64_t &download_file_size) {
  return error_s::make();
}

Under this impl you cannt print logs in do_download, however, you can return an error with string (error_s) in failure.

error_code replica::do_download(const std::string &remote_dir,
                                const std::string &local_dir,
                                const std::string &file_name,
                                dist::block_service::block_filesystem *fs,
                                /*out*/ uint64_t &download_file_size) {
  error_s err = download_remote_files();
  if (!err.is_ok())
    derror_replica(err);
  } else {
    ddebug_replica("download succeed with total file size {}", download_file_size);
  }
}

[hycdong]

I think current implementation is during a middle stage, it is okay to leave do_download function in replica namespace.
If I move download function into block_service module, I should also move unit tests into block_service module. However, cold backup has wrote a block_file_mock in replica unit test for backup unit tests. If I move it into block_service, I should update backup unit tests code, which will be refactored in further pull request finally, it is not meaningful to refactor them currently.
In my view, we should move all functions related to file provider into block_service module, such as download files, upload files. It can be implemented after refactoring backup and restore code.

[hycdong]

As we discuss offline, I will refactor code in next pull request.