Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add gcs scanning integration #1153

Merged
merged 78 commits into from
Mar 8, 2023
Merged

Add gcs scanning integration #1153

merged 78 commits into from
Mar 8, 2023

Conversation

ahrav
Copy link
Collaborator

@ahrav ahrav commented Mar 4, 2023 •
edited
Loading

  • Add source integration for scanning GCS
  • There are multiple ways to provide auth to run the scan the recommend way is using ADC

Screenshot 2023-03-07 at 5 12 03 PM

Note: The notion of "resuming" is currently supported by the gcsManager, the logic still needs to be added to the gcs source. As this PR is already very large i'll add it in a follow up PR.

ahrav added 30 commits March 1, 2023 16:39
@ahrav
Setup for GCS scanning.
862d8e0
@ahrav
Update GCS engine w/ projectID req.
1c894a9
@ahrav
Add concurrency field to gcsManager.
979f1e3
@ahrav
add errgroup to gcsManager.
e85e005
@ahrav
Update gcs manager.
7867306
@ahrav
Use defautl ADC.
6c068b6
@ahrav
use ADC.'
ab268a9
@ahrav
Add TOOD.
30c6a80
@ahrav
add log to iterator completion.
57ec105
@ahrav
use a BinaryReader instead of concrete object for channel type.
e71d017
@ahrav
initial test for Chunks.
63a8ccf
@ahrav
Add tests for chunking objects.
34ab6ff
@ahrav
Add concurrency.
b592f98
@ahrav
update metadata to include content type and acls.
b411486
@ahrav
Add object reading code.
b842e10
@ahrav
Add integration test.
e284a7c
@ahrav
Add entrypoint.
98f2081
@ahrav
Add removed wg.Wait().
3e974e7
@ahrav
remove dead code.
7a5534f
@ahrav
remove build.
7959b0f
@ahrav
Remove period from file extension.
a087c7b
@ahrav
remove used.
dd91012
@ahrav
Add comment.
b1159f3
@ahrav
Setup for GCS scanning.
4ca2aec
@ahrav
Update GCS engine w/ projectID req.
d3dad17
@ahrav
Add concurrency field to gcsManager.
bff69e5
@ahrav
add errgroup to gcsManager.
0de8dac
@ahrav
Update gcs manager.
35839a0
@ahrav
Use defautl ADC.
78100a3
@ahrav
use ADC.'
cd6da2d
@ahrav ahrav marked this pull request as ready for review March 7, 2023 23:58
@ahrav ahrav requested review from a team as code owners March 7, 2023 23:58
@ahrav ahrav merged commit cbf299a into main Mar 8, 2023
@ahrav ahrav deleted the add-gcs-scanning branch March 8, 2023 01:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dustin-decker dustin-decker dustin-decker approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ahrav @dustin-decker