-
Notifications
You must be signed in to change notification settings - Fork 772
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
object_store: support encoded path as input #3651
Comments
Have you tried using |
@tustvold yes, I have tried. I understand that However, when client GET object, it will re-encode at L246 -> L211 -> L459 below arrow-rs/object_store/src/aws/client.rs Lines 237 to 246 in 2b9bbce
arrow-rs/object_store/src/aws/client.rs Lines 209 to 213 in 2b9bbce
arrow-rs/object_store/src/aws/client.rs Lines 458 to 460 in 2b9bbce
I imagine L211 need to know whether input |
Yes, this is so that the path of the created object matches exactly the
What is percent encoding this input, if you call |
yes, it work
this is from ROAPI config, the user input can be already percent encoded. ROAPI currently decode it before passing to |
Adding a This would likely still need to decode the path, in order to ensure path safety, but would save users that complexity. FWIW overheads of encoding and signing the requests will dominate any additional latency resulting from this encoding dance, and both will be completely dominated by the latency of the store itself (typically in the 10s or 100s of milliseconds). |
thank you, I have draft the PR here: #3663, waiting CI to start |
|
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
while passing an percentage encoded Path to object store, it re-encode before get object.
e.g.
S3
arrow-rs/object_store/src/aws/client.rs
Line 459 in 2b9bbce
GCS
arrow-rs/object_store/src/gcp/mod.rs
Line 270 in 2b9bbce
a workaround is to pass decoded path from upstream application, even though upstream already receive encoded path.
e.g. roapi/roapi#98 (comment)
Describe the solution you'd like
ideally, path should just need encode once. One idea is to improve
Path
to include new attribute called 'percent_encoded'arrow-rs/object_store/src/path/mod.rs
Lines 135 to 138 in 2b9bbce
Describe alternatives you've considered
auto detect whether already percentage encode?
Additional context
The text was updated successfully, but these errors were encountered: