object_store: support encoded path as input

[jychen7]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
while passing an percentage encoded Path to object store, it re-encode before get object.
e.g.

S3

arrow-rs/object_store/src/aws/client.rs

Line 459 in 2b9bbce

utf8_percent_encode(path.as_ref(), &STRICT_PATH_ENCODE_SET)

GCS

arrow-rs/object_store/src/gcp/mod.rs

Line 270 in 2b9bbce

percent_encoding::utf8_percent_encode(path.as_ref(), NON_ALPHANUMERIC);

a workaround is to pass decoded path from upstream application, even though upstream already receive encoded path.

e.g. roapi/roapi#98 (comment)

Describe the solution you'd like
ideally, path should just need encode once. One idea is to improve Path to include new attribute called 'percent_encoded'

arrow-rs/object_store/src/path/mod.rs

Lines 135 to 138 in 2b9bbce

	pub struct Path {
	/// The raw path with no leading or trailing delimiters
	raw: String,
	}

Describe alternatives you've considered
auto detect whether already percentage encode?

Additional context

[tustvold]

Have you tried using Path::parse, this shouldn't re-encode

[jychen7]

@tustvold yes, I have tried.

I understand that Path::parse("blogs%20space.parquet") will NOT re-encode, the result is Path { raw: "blogs%20space.parquet" }.

However, when client GET object, it will re-encode at L246 -> L211 -> L459 below

arrow-rs/object_store/src/aws/client.rs

Lines 237 to 246 in 2b9bbce

	pub async fn get_request(
	&self,
	path: &Path,
	range: Option<Range<usize>>,
	head: bool,
	) -> Result<Response> {
	use reqwest::header::RANGE;
	let credential = self.get_credential().await?;
	let url = self.config.path_url(path);

arrow-rs/object_store/src/aws/client.rs

Lines 209 to 213 in 2b9bbce

	impl S3Config {
	fn path_url(&self, path: &Path) -> String {
	format!("{}/{}", self.bucket_endpoint, encode_path(path))
	}
	}

arrow-rs/object_store/src/aws/client.rs

Lines 458 to 460 in 2b9bbce

	fn encode_path(path: &Path) -> PercentEncode<'_> {
	utf8_percent_encode(path.as_ref(), &STRICT_PATH_ENCODE_SET)
	}

I imagine L211 need to know whether input Path is already encoded, either via new attribute or auto detect

[tustvold]

However, when client GET object, it will re-encode at L246 -> L211 -> L459 below

Yes, this is so that the path of the created object matches exactly the Path provided. So if you provide blogs%20space.parquet that is what will be created in object storage. This is the same as if you provided blogs%invalidescape.parquet.

Path::parse("blogs%20space.parquet")

What is percent encoding this input, if you call Path::parse("blogs space.parquet") everything should work? Is it possible something is receiving a URL and not decoding it properly, and just handing off the raw path?

[jychen7]

if you call Path::parse("blogs space.parquet") everything should work?

yes, it work

What is percent encoding this input
Is it possible something is receiving a URL and not decoding it properly, and just handing off the raw path?

this is from ROAPI config, the user input can be already percent encoded. ROAPI currently decode it before passing to object_store. I am wondering whether object_store can support receiving encoded path, so from end to end, we don't need additional decode -> encode
More detail here: roapi/roapi#98 (comment)

[tustvold]

can support receiving encoded path

Adding a Path::from_url_path that accepts a Url argument makes sense to me, I would be happy to review a PR that adds this 👍

This would likely still need to decode the path, in order to ensure path safety, but would save users that complexity.

FWIW overheads of encoding and signing the requests will dominate any additional latency resulting from this encoding dance, and both will be completely dominated by the latency of the store itself (typically in the 10s or 100s of milliseconds).

[jychen7]

I would be happy to review a PR that adds this

thank you, I have draft the PR here: #3663, waiting CI to start

[tustvold]

label_issue.py automatically added labels {'object-store'} from #3663