Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kasiahinkson/extend list blobs #953

Merged
merged 40 commits into from
Dec 11, 2023
Merged
Changes from 39 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
ee4ec8b
Merge main into major-release (#814)
Jason94 May 8, 2023
f694b4c
Merge branch 'major-release' of https://github.com/move-coop/parsons …
May 9, 2023
33b9050
Merge branch 'main' into major-release
Aug 4, 2023
de08af5
DatabaseConnector Interface to Major Release (#815)
Jason94 Aug 17, 2023
f78297c
Zoom Authentication + Polling API (#873)
IanRFerguson Aug 29, 2023
7ad3036
Merging Main Before Release (#880)
IanRFerguson Aug 29, 2023
29e9f3e
Switch from API key to Personal Access Token (#866)
crayolakat Sep 7, 2023
f6ee530
merge main
willyraedy Sep 22, 2023
55d3717
Wraedy/bigquery db connector (#875)
willyraedy Oct 6, 2023
b562420
BigQuery - Add Column Helpers (#911)
IanRFerguson Oct 30, 2023
b5342c9
flip default params
IanRFerguson Oct 30, 2023
16c40a3
flip back
IanRFerguson Oct 30, 2023
5efd71e
Google BigQuery - Clean Up Autodetect Logic (#914)
IanRFerguson Nov 2, 2023
400fa39
Update stale references to parsons.databases.bigquery (#920)
dexchan Nov 13, 2023
0e2d848
Fix gcs hidden error (#930)
sharinetmc Nov 30, 2023
c347777
Allow list blobs to return all info
KasiaHinkson Dec 1, 2023
b5ffc22
Return list
KasiaHinkson Dec 1, 2023
89f51f1
List comprehension awesomeness
KasiaHinkson Dec 1, 2023
231d16b
trying something
KasiaHinkson Dec 1, 2023
17ff564
trying
KasiaHinkson Dec 1, 2023
6f97803
Moving things
KasiaHinkson Dec 1, 2023
1f39ad1
trying another thing
KasiaHinkson Dec 1, 2023
3d1663d
return all blobs
IanRFerguson Dec 1, 2023
a3b27d4
Forgot to change param back
KasiaHinkson Dec 1, 2023
8b843f2
GoogleCloudStorage - Handle zip / gzip files flexibly (#937)
IanRFerguson Dec 1, 2023
e1a7cce
GoogleCloudStorage - Add GCS Destination Path Param (#936)
IanRFerguson Dec 1, 2023
ab4b555
Merge branch 'main' into major-release
shaunagm Dec 1, 2023
d812efc
Bump version number to 3.0.0
shaunagm Dec 1, 2023
9fe9a85
Fix import statement in test_bigquery.py
shaunagm Dec 1, 2023
5a9f87e
Add Tests to major-release Branch (#949)
IanRFerguson Dec 7, 2023
d1848c1
Resolve GCP Test Failures For Major Release (#948)
IanRFerguson Dec 8, 2023
3f70a6c
Install google-cloud-storage-transfer for google extras (#946)
austinweisgrau Dec 8, 2023
80210fe
Revert "Enable passing `identifiers` to ActionNetwork `upsert_person(…
austinweisgrau Dec 8, 2023
41cd255
BigQuery - Add row count function to connector (#913)
IanRFerguson Dec 8, 2023
5e528d6
add examples (#952)
IanRFerguson Dec 8, 2023
afca41c
Merge branch 'main' into major-release
shaunagm Dec 8, 2023
14df184
Parse Boolean types by default (#943)
austinweisgrau Dec 8, 2023
a679731
Merge branch 'major-release' into kasiahinkson/extend-list-blobs
IanRFerguson Dec 11, 2023
e9d324f
Merge branch 'main' into kasiahinkson/extend-list-blobs
IanRFerguson Dec 11, 2023
08b06d4
address comments
IanRFerguson Dec 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion parsons/google/google_cloud_storage.py
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,14 @@ def delete_bucket(self, bucket_name, delete_blobs=False):
bucket.delete(force=delete_blobs)
logger.info(f"{bucket_name} bucket deleted.")

def list_blobs(self, bucket_name, max_results=None, prefix=None, match_glob=None):
def list_blobs(
self,
bucket_name,
max_results=None,
prefix=None,
match_glob=None,
include_file_details=False,
Jason94 marked this conversation as resolved.
Show resolved Hide resolved
):
"""
List all of the blobs in a bucket

Expand All @@ -158,6 +165,10 @@ def list_blobs(self, bucket_name, max_results=None, prefix=None, match_glob=None
blobs = self.client.list_blobs(
bucket_name, max_results=max_results, prefix=prefix, match_glob=match_glob
)

if include_file_details:
return [b for b in blobs]

lst = [b.name for b in blobs]
logger.info(f"Found {len(lst)} in {bucket_name} bucket.")
Jason94 marked this conversation as resolved.
Show resolved Hide resolved

Expand Down