Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Donate object_store code from object_store_rs to arrow-rs #2081

Merged
merged 15 commits into from
Jul 22, 2022

Conversation

alamb
Copy link
Contributor

@alamb alamb commented Jul 15, 2022

Draft until

Which issue does this PR close?

re #2030

Rationale for this change

See #2030

What changes are included in this PR?

These are kept in different commits for easier review:

Follow on Items:

Are there any user-facing changes?

Not yet

@tustvold
Copy link
Contributor

I wonder if we could preserve the history somehow, I'm mainly interested in not losing releases, but this would also help with in-flight PRs.

Perhaps you could add object_store_rs as a remote, add a commit moving it to a subdirectory, and then merge it into arrow-rs?

@alamb
Copy link
Contributor Author

alamb commented Jul 15, 2022

Update here: my git-fu is not strong enough to try and keep the history from object_store_rs -- @tustvold plans to give it a try. In the interim I will work on porting the tests to use github actions

@tustvold tustvold mentioned this pull request Jul 15, 2022
@codecov-commenter
Copy link

codecov-commenter commented Jul 15, 2022

Codecov Report

Merging #2081 (01f3908) into master (34f58f9) will decrease coverage by 0.78%.
The diff coverage is 55.08%.

@@            Coverage Diff             @@
##           master    #2081      +/-   ##
==========================================
- Coverage   83.71%   82.93%   -0.79%     
==========================================
  Files         225      237      +12     
  Lines       59632    61322    +1690     
==========================================
+ Hits        49923    50855     +932     
- Misses       9709    10467     +758     
Impacted Files Coverage Δ
object_store/src/aws.rs 0.00% <0.00%> (ø)
object_store/src/azure.rs 0.00% <0.00%> (ø)
object_store/src/gcp.rs 0.00% <0.00%> (ø)
object_store/src/oauth.rs 0.00% <0.00%> (ø)
object_store/src/token.rs 0.00% <0.00%> (ø)
object_store/src/util.rs 50.00% <50.00%> (ø)
object_store/src/local.rs 82.88% <82.88%> (ø)
object_store/src/lib.rs 84.42% <84.42%> (ø)
object_store/src/path/mod.rs 90.90% <90.90%> (ø)
object_store/src/memory.rs 92.17% <92.17%> (ø)
... and 5 more

Help us with your feedback. Take ten seconds to tell us how you rate us.

@tustvold
Copy link
Contributor

keep the history from object_store_rs

So unfortunately object_store_rs contains merge commits, so the only way to do this would break the linear history of arrow-rs, either by keeping the merge commits from object_store_rs/main or by adding a single merge commit. I therefore think its not worth it.

@alamb
Copy link
Contributor Author

alamb commented Jul 16, 2022

Update on test migration:

  1. There appears to be one test that needs to be rewritten for github runners https://github.com/apache/arrow-rs/runs/7362535940?check_suite_focus=true (bubble_up_io_errors)
  2. I have gotten the emulator tests running using github actions here: Add github test skeleton (NOT FOR MERGING) influxdata/object_store_rs#51 (it needs some polishing but I propose merging this PR first and then adding in object_store tests after the merge

@alamb
Copy link
Contributor Author

alamb commented Jul 18, 2022

There appears to be one test that needs to be rewritten for github runners https://github.com/apache/arrow-rs/runs/7362535940?check_suite_focus=true (bubble_up_io_errors)

@tustvold sorry to bother you -- but I was wondering if you had any ideas of ways to port the above test to run on github actions -- specifically it appears that setting the file mode to 0o000 doesn't actually generate an Error 🤔

@tustvold
Copy link
Contributor

tustvold commented Jul 18, 2022

Probably the Github Actions runner is running as root, and therefore the permission failure doesn't occur. I'll rework the test

Edit: I can't think of an obvious way around this, I'd just remove the test - it isn't immensely valuable

@alamb
Copy link
Contributor Author

alamb commented Jul 20, 2022

Probably the Github Actions runner is running as root, and therefore the permission failure doesn't occur. I'll rework the test

I have #[ignored] the test in ed0f926 and will file a follow on ticket to fix it

@alamb alamb marked this pull request as ready for review July 21, 2022 21:15
@alamb alamb merged commit 866f1a1 into apache:master Jul 22, 2022
@alamb alamb deleted the alamb/object_store_rs branch July 22, 2022 18:30
@ursabot
Copy link

ursabot commented Jul 22, 2022

Benchmark runs are scheduled for baseline = add2649 and contender = 866f1a1. 866f1a1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants