Update BigQuery connector documentation #25109

SemionPar · 2025-02-21T16:47:34Z

Description

Add a section on project ID resolution.

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

## Section
* Fix some things. ({issue}`issuenumber`)

SemionPar · 2025-02-21T17:34:57Z

I have one more thing I am considering adding here.

Use case - cross project service accounts, aspect - PTF functions.

Little on the use case and how to setup service accounts in this way:

https://cloud.google.com/eventarc/docs/use-cross-project-service-account (in context on another GCP service, but seems relevant)

By default, you can't create an IAM service account in one Google Cloud project and attach it to a resource in another project. However, you might have centralized the service accounts for your organization in separate projects, which can make the service accounts easier to manage. This document outlines the steps required to support attaching a service account in one project to an Eventarc trigger in another project.

https://www.cloudquery.io/blog/creating-cross-project-service-accounts-in-gcp

Illustration

Setup

I did some testing with 3 Google Cloud projects

Projects:

base-30275 - 'parent' project
data-1-13082, data-2-644 - 'data' projects

Service accounts:

base-30275-service-account-34@base-30275.iam.gserviceaccount.com
- lives in base-30275,
- has BigQueryUser access to base-30275, data-2-644
base-30275-no-bq@base-30275.iam.gserviceaccount.com
- lives in base-30275,
- has BigQueryUser access to data-2-644 and data-1-13082, BUT NOT TO base-30275,
data-2-bq@data-2-644.iam.gserviceaccount.com
- lives in data-2-644,
- has BigQueryUser access to data-2-644

Findings

PTF query will execute with service account permissions, regardless of the bigquery.parent-project-id or bigquery.project-id

This makes it possible to access ALL BigQuery resources that this service account has permissions to:

-Dtesting.bigquery.parent-project-id=data-2-644         <--------------- data project id set as parent project
-Dtesting.bigquery.project-id=data-2-644
-Dtesting.bigquery.credentials-key=base-30275-service-account-34@base-30275.iam.gserviceaccount.com

-- access to data project:
  SELECT * FROM TABLE(bigquery.system.query(query => 'SELECT schema_name FROM `data-2-644.region-us.INFORMATION_SCHEMA.SCHEMATA`'));
-- access to parent project:
  SELECT * FROM TABLE(bigquery.system.query(query => 'SELECT schema_name FROM `base-30275.region-us.INFORMATION_SCHEMA.SCHEMATA`'));
-- OK                                                   <--------------- has access because service account has permissions in parent
-- access to another data project:
  SELECT * FROM TABLE(bigquery.system.query(query => 'SELECT schema_name FROM `data-1-13082.region-us.INFORMATION_SCHEMA.SCHEMATA`'));
-- OK                                                   <--------------- has access because service account has permissions in other data project as well

If, however, used with service account that has no BigQuery permissions in the parent project, access is denied:

-Dtesting.bigquery.parent-project-id=data-2-644
-Dtesting.bigquery.project-id=data-2-644
-Dtesting.bigquery.credentials-key=base-30275-service-account-34@base-30275.iam.gserviceaccount.com

-- access to data project:
  SELECT * FROM TABLE(bigquery.system.query(query => 'SELECT schema_name FROM `data-2-644.region-us.INFORMATION_SCHEMA.SCHEMATA`'));
-- access to parent project:
  SELECT * FROM TABLE(bigquery.system.query(query => 'SELECT schema_name FROM `base-30275.region-us.INFORMATION_SCHEMA.SCHEMATA`'));
-- Failed to get destination table for query. Access Denied: Table base-30275:region-us. INFORMATION_SCHEMA. SCHEMATA: User does not have permission to query table base-30275:region-us. INFORMATION_SCHEMA. SCHEMATA, or perhaps it does not exist.

Summary

If a BigQuery catalog is configured with an SA JSON key and a project ID, then through PTF, one effectively gains access to all BigQuery projects that the service account has permissions for.

Ultimately, do we want to mention this in the documentation?

hashhar

Great job demystifying this.

Everyone was so confused for so long and wasn't able to figure out how or why it works.

docs/src/main/sphinx/connector/bigquery.md

mosabua

Overall we might want to rename the section since it seems all about billing ... also since it seems pretty critical we might want to add an anchor and link to it from the configs for each of these properties with a "see also .."

docs/src/main/sphinx/connector/bigquery.md

mosabua · 2025-02-25T17:09:17Z

docs/src/main/sphinx/connector/bigquery.md

 data in multiple GCP projects, You need to create several catalogs, each
 pointing to a different GCP project. For example, if you have two GCP projects,
 one for the sales and one for analytics, you can create two properties files in
 `etc/catalog` named `sales.properties` and `analytics.properties`, both
 having `connector.name=bigquery` but with different `project-id`. This will
 create the two catalogs, `sales` and `analytics` respectively.

+### Understanding Project ID Resolution
+
+The BigQuery connector determines the project ID to use based on the 


Somewhere we need to link to some sort docs from BigQuery that explains more about the project ID ideally .. maybe right here

Fine idea! I am having trouble finding any relevant documentation on this topic that is more than a brief mention in the BigQuery section of Google Cloud docs. There are some third party sources like https://productresources.collibra.com/docs/collibra/latest/Content/DataQuality/DBConnection/ta_bigquery-cross-account-dataset-access.htm

We could have a reference to a general cross project SA page like this one instead, WDYT? https://cloud.google.com/iam/docs/attach-service-accounts#attaching-different-project

Maybe use this link https://cloud.google.com/resource-manager/docs/creating-managing-projects .. it explicitly explains Project ID

I went ahead and added a sentence, please edit as you see fit

Link the project ID to the link I found

docs/src/main/sphinx/connector/bigquery.md

mosabua · 2025-02-25T17:15:46Z

With regards to the query table function and it being a potential security issue.. thats already documented in a generic fashion .. adding more details specific to BigQuery would also be good.

SemionPar · 2025-02-25T18:27:20Z

Thank you for reviews @hashhar @mosabua!

Applied the changes and pushed, PTAL

docs/src/main/sphinx/connector/bigquery.md

SemionPar · 2025-02-25T21:50:41Z

Thank you @mosabua!

Gave it another try, PTAL

hashhar

LGTM % nits

docs/src/main/sphinx/connector/bigquery.md

SemionPar · 2025-02-26T09:42:14Z

Thank you @mosabua!

Gave it another try, PTAL

Sorry, forgot to actually push my latest changes yesterday... All pushed now

mosabua

A few nits .. then ready to go ... ping me after last push and I can merge

mosabua · 2025-02-26T19:10:25Z

docs/src/main/sphinx/connector/bigquery.md

-The BigQuery connector can only access a single GCP project.Thus, if you have
-data in multiple GCP projects, You need to create several catalogs, each
+The BigQuery connector can only access a single GCP project. Thus, if you have
+data in multiple GCP projects, you need to create several catalogs, each


Suggested change

data in multiple GCP projects, you need to create several catalogs, each

data in multiple GCP projects, you must create several catalogs, each

mosabua · 2025-02-26T19:10:44Z

docs/src/main/sphinx/connector/bigquery.md

@@ -74,14 +74,47 @@ bigquery.project-id=<your Google Cloud Platform project id>

 ### Multiple GCP projects

-The BigQuery connector can only access a single GCP project.Thus, if you have
-data in multiple GCP projects, You need to create several catalogs, each
+The BigQuery connector can only access a single GCP project. Thus, if you have


Suggested change

The BigQuery connector can only access a single GCP project. Thus, if you have

The BigQuery connector can only access a single GCP project. If you have

mosabua · 2025-02-26T19:11:30Z

docs/src/main/sphinx/connector/bigquery.md

+(bigquery-project-id-resolution)=
+### Billing and data projects
+
+The BigQuery connector determines the


Fix wrapping in this paragraph to 80 char .. currently its weird

mosabua · 2025-02-26T19:11:51Z

docs/src/main/sphinx/connector/bigquery.md

+[project ID](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
+to use based on the configuration settings.
+This behavior provides users with flexibility in selecting both
+the project to query and the project to be billed for BigQuery operations.


Suggested change

the project to query and the project to be billed for BigQuery operations.

the project to query and the project to bill for BigQuery operations.

findinpath · 2025-02-26T19:14:15Z

@mosabua pls go ahead and change the wording yourself - seems straightforward.

mosabua · 2025-02-26T19:16:33Z

@mosabua pls go ahead and change the wording yourself - seems straightforward.

I don't have time for that until Friday

cla-bot bot added the cla-signed label Feb 21, 2025

github-actions bot added the docs label Feb 21, 2025

findinpath requested review from mosabua and findinpath February 24, 2025 15:32

anusudarsan requested a review from Ordinant February 25, 2025 16:51

hashhar reviewed Feb 25, 2025

View reviewed changes

docs/src/main/sphinx/connector/bigquery.md Outdated Show resolved Hide resolved

mosabua requested changes Feb 25, 2025

View reviewed changes

SemionPar force-pushed the semionpar/update-bigquery-connector-documentation branch 2 times, most recently from 164193f to c71334f Compare February 25, 2025 18:24

SemionPar changed the title ~~[wip] Update BigQuery connector documentation~~ Update BigQuery connector documentation Feb 25, 2025

SemionPar marked this pull request as ready for review February 25, 2025 18:25

SemionPar requested review from mosabua and hashhar February 25, 2025 18:27

mosabua reviewed Feb 25, 2025

View reviewed changes

docs/src/main/sphinx/connector/bigquery.md Outdated Show resolved Hide resolved

mosabua requested changes Feb 25, 2025

View reviewed changes

SemionPar requested a review from mosabua February 25, 2025 21:50

anusudarsan approved these changes Feb 26, 2025

View reviewed changes

hashhar approved these changes Feb 26, 2025

View reviewed changes

docs/src/main/sphinx/connector/bigquery.md Outdated Show resolved Hide resolved

docs/src/main/sphinx/connector/bigquery.md Outdated Show resolved Hide resolved

SemionPar added 2 commits February 26, 2025 10:38

Fix typo

05bcef8

Document BigQuery connector project ID resolution

211a2f1

SemionPar force-pushed the semionpar/update-bigquery-connector-documentation branch from 3758f98 to 211a2f1 Compare February 26, 2025 09:41

mosabua requested changes Feb 26, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update BigQuery connector documentation #25109

Update BigQuery connector documentation #25109

SemionPar commented Feb 21, 2025

SemionPar commented Feb 21, 2025 •

edited

Loading

hashhar left a comment

mosabua left a comment

mosabua Feb 25, 2025

SemionPar Feb 25, 2025

mosabua Feb 25, 2025 •

edited

Loading

SemionPar Feb 25, 2025

mosabua Feb 25, 2025

mosabua commented Feb 25, 2025

SemionPar commented Feb 25, 2025

SemionPar commented Feb 25, 2025

hashhar left a comment

SemionPar commented Feb 26, 2025

mosabua left a comment

mosabua Feb 26, 2025

mosabua Feb 26, 2025

mosabua Feb 26, 2025

mosabua Feb 26, 2025

findinpath commented Feb 26, 2025

mosabua commented Feb 26, 2025

	data in multiple GCP projects, you need to create several catalogs, each
	data in multiple GCP projects, you must create several catalogs, each

	The BigQuery connector can only access a single GCP project. Thus, if you have
	The BigQuery connector can only access a single GCP project. If you have

	the project to query and the project to be billed for BigQuery operations.
	the project to query and the project to bill for BigQuery operations.

Update BigQuery connector documentation #25109

Are you sure you want to change the base?

Update BigQuery connector documentation #25109

Conversation

SemionPar commented Feb 21, 2025

Description

Additional context and related issues

Release notes

SemionPar commented Feb 21, 2025 • edited Loading

Illustration

Setup

Findings

Summary

hashhar left a comment

Choose a reason for hiding this comment

mosabua left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mosabua Feb 25, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mosabua commented Feb 25, 2025

SemionPar commented Feb 25, 2025

SemionPar commented Feb 25, 2025

hashhar left a comment

Choose a reason for hiding this comment

SemionPar commented Feb 26, 2025

mosabua left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findinpath commented Feb 26, 2025

mosabua commented Feb 26, 2025

SemionPar commented Feb 21, 2025 •

edited

Loading

mosabua Feb 25, 2025 •

edited

Loading