Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cfg): Add configurable GCP log page size #3556

Merged
merged 7 commits into from
Nov 29, 2021
Merged

feat(cfg): Add configurable GCP log page size #3556

Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5fb159d
Add configurable log page size
jjoyce0510 Nov 11, 2021
0653b49
Update type annotation
jjoyce0510 Nov 11, 2021
960bdd9
Add Pydantic integer
jjoyce0510 Nov 11, 2021
8187c30
Remove unnecessary file
jjoyce0510 Nov 11, 2021
a54084c
spacing
jjoyce0510 Nov 11, 2021
2d79d51
Make log size configurable
jjoyce0510 Nov 22, 2021
c5d065e
fix lintey
jjoyce0510 Nov 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions metadata-ingestion/src/datahub/ingestion/source/usage/bigquery_usage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@
AuditLogEntry = Any

DEBUG_INCLUDE_FULL_PAYLOADS = False
GCP_LOGGING_PAGE_SIZE = 1000

# Handle yearly, monthly, daily, or hourly partitioning.
# See https://cloud.google.com/bigquery/docs/partitioned-tables.
Expand Down Expand Up @@ -257,6 +256,7 @@ class BigQueryUsageConfig(BaseUsageConfig):
extra_client_options: dict = {}
env: str = builder.DEFAULT_ENV

log_page_size: pydantic.PositiveInt = 1000
query_log_delay: Optional[pydantic.PositiveInt] = None
max_query_duration: timedelta = timedelta(minutes=15)

Expand Down Expand Up @@ -342,7 +342,7 @@ def get_entry_timestamp(entry: AuditLogEntry) -> datetime:
for client in clients:
try:
list_entries: Iterable[AuditLogEntry] = client.list_entries(
filter_=filter, page_size=GCP_LOGGING_PAGE_SIZE
filter_=filter, page_size=self.config.log_page_size
)
list_entry_generators_across_clients.append(list_entries)
except Exception as e:
Expand Down