Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Go SDK: GCS FS supports rename, improved test coverage. Local FS supports copy. #26065

Merged
merged 1 commit into from
Apr 11, 2023
Merged

Conversation

jeremyje
Copy link
Contributor

@jeremyje jeremyje commented Apr 1, 2023

Fixes #26064

All changes are for the Go SDK.

  • Add support for renaming GCS files. This is performed by a copy and then delete so it is not atomic. The Java implementation follows this approach.
  • Add support for Copy in Local FileSystem.
  • Add test coverage for the GCS FileSystem implementation. This adds a new test dependency to github.com/fsouza/fake-gcs-server/fakestorage.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • [?] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI.

@codecov
Copy link

codecov bot commented Apr 1, 2023

Codecov Report

Merging #26065 (850549e) into master (5558859) will increase coverage by 0.03%.
The diff coverage is 53.12%.

@@            Coverage Diff             @@
##           master   #26065      +/-   ##
==========================================
+ Coverage   71.18%   71.21%   +0.03%     
==========================================
  Files         787      787              
  Lines      103298   103330      +32     
==========================================
+ Hits        73528    73588      +60     
+ Misses      28288    28245      -43     
- Partials     1482     1497      +15     
Flag Coverage Δ
go 53.82% <53.12%> (+0.12%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...dks/python/apache_beam/options/pipeline_options.py 94.02% <ø> (ø)
sdks/go/pkg/beam/io/filesystem/gcs/gcs.go 59.45% <52.63%> (+47.50%) ⬆️
sdks/go/pkg/beam/io/filesystem/local/local.go 59.52% <53.84%> (-2.55%) ⬇️

... and 2 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@github-actions
Copy link
Contributor

github-actions bot commented Apr 1, 2023

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @damccorm for label go.
R: @pabloem for label io.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@damccorm
Copy link
Contributor

damccorm commented Apr 3, 2023

assign to next reviewer

@github-actions
Copy link
Contributor

github-actions bot commented Apr 3, 2023

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @jrmccluskey for label go.
R: @pabloem for label io.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@jrmccluskey
Copy link
Contributor

Run GoPortable PreCommit

sdks/go/pkg/beam/io/filesystem/gcs/gcs_test.go Outdated Show resolved Hide resolved
sdks/go/pkg/beam/io/filesystem/gcs/gcs_test.go Outdated Show resolved Hide resolved
@jrmccluskey
Copy link
Contributor

Run Go PreCommit

@jrmccluskey
Copy link
Contributor

Run GoPortable PreCommit

@jrmccluskey
Copy link
Contributor

Run RAT PreCommit

@jrmccluskey
Copy link
Contributor

Run Whitespace PreCommit

Copy link
Contributor

@jrmccluskey jrmccluskey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM mod the change to CHANGES.md. I'll merge when that's updated. Thank you!

CHANGES.md Outdated Show resolved Hide resolved
@jeremyje
Copy link
Contributor Author

Updated the CHANGES.md file.

@jeremyje
Copy link
Contributor Author

Rebased with latest changes and reran go mod tidy since there were conflicts.

@jrmccluskey
Copy link
Contributor

I don't think Jenkins picked up the precommits for some reason, let me see if I can get it to run...

@jrmccluskey
Copy link
Contributor

retest this please

@jrmccluskey
Copy link
Contributor

Thank you for your contribution and your patience while we got everything tested!

@jrmccluskey jrmccluskey merged commit 78957df into apache:master Apr 11, 2023
@jeremyje jeremyje deleted the gcsrename branch April 11, 2023 20:07
lostluck pushed a commit to lostluck/beam that referenced this pull request Apr 12, 2023
jakubrauch pushed a commit to jakubrauch/beam that referenced this pull request Apr 27, 2023
rezarokni pushed a commit to rezarokni/beam that referenced this pull request May 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request]: Go SDK GCS FileSystem does not support rename.
3 participants