[Feature Request]: Add Google Drive scope to GCP Auth

[ricardograca-scratch]

What would you like to happen?

The current Python SDK doesn't include the https://www.googleapis.com/auth/drive scope when sending the authentication request to Google Cloud. This means that it's not possible to access Google Drive backed tables in BigQuery.

Adding the scope does indeed provide the required access. I'm only interested in the Python SDK, but this change should be easy to apply on all other SDKs as well if there's interest in that.

I can work on this change and submit a pull request.

Issue Priority

Priority: 2

Issue Component

Component: sdk-py-core

[tvalentyn]

What happens if Drive API is not enabled for a project but we include it with a scope?

[ricardograca-scratch]

I imagine the same thing that happens with any other scope, but I'll check and report back.

[ricardograca-scratch]

@tvalentyn The Drive API being enabled or not has not effect on the result if the extra scope is included. However, without the scope, even if the file is shared with the service account accessing the data this error is displayed and it's not possible to access the table:

Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials.

Adding the scope resolves the issue.

[tvalentyn]

Thanks for checking.

Would adding https://www.googleapis.com/auth/cloud-platform also fix the problem or this specific drive scope is necessary?

[tvalentyn]

If not, I would suggest adding a new pipeline option that allows such customizations.

Looks like in Java it's possible to specify a custom credential via a factory or credential instance : https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/extensions/gcp/options/GcpOptions.html . In Python we don't have this. But perhaps passing a string of custom scopes to use when authenticating would be sufficient?

[ricardograca-scratch]

Would adding https://www.googleapis.com/auth/cloud-platform also fix the problem or this specific drive scope is necessary?

It's already there and it doesn't allow access to Google Drive backed tables, so the drive scope is necessary.

If not, I would suggest adding a new pipeline option that allows such customizations.

No other scope is treated this way, and it seems like I would need to do a lot of modifications to beam to enable that. I'm no beam expert, not to mention I have no experience developing Java or Go, so the likelihood of me actually pulling off what you propose is very slim, since it would require a substantial amount of time.

Can't I just add the necessary scope since it doesn't seem to cause any issues?

[kennknowles]

It seems to me that your proposed change is consistent with the current design so we could go ahead as a workaround. But at the same time the current design is odd, since it adds a totally arbitrary set of scopes instead of just the necessary set, so we should fix that.

[ricardograca-scratch]

I agree with that. The ideal solution for me would be to provide a minimal set that is required to access BigQuery and allow users to specify any extra scopes needed.

I'll work on this change as it's currently proposed for now.

[lukecwik]

I disagree that adding scopes is a good idea, we should instead allow users to configure the scopes themselves. Adding a credential factory to python or adding a pipeline option that enables configuring these scopes with the default being the currently enabled list makes a lot more sense to me.

[kennknowles]

I agree with your comment. I am separating the immediate need with the longer term refactor. Currently, we have an arbitrary set of scopes that probably never should have been done this way. This proposal adds one, keeping the design as-is.

Refactoring the design to use only minimal scopes specified by the user, or perhaps by the IOs in the pipeline, is still the best way to go in the future.

[lukecwik]

I would disagree, having existing users who upgrade to this version pick up additional scopes that they may not have been aware of is not a good experience.

[kennknowles]

Again, I agree with you, but that is status quo. That is exactly what happened when SpannerIO was added.

[kennknowles]

If it is easy to do The Right Thing then go ahead and do it. But forcing arbitrary discipline on new contributors that we ourselves never complied with is not how we should run things.

[lukecwik]

Continuing to do something that was done incorrectly in the past and digging a bigger hole for future maintenance is a problem.

[tvalentyn]

That is exactly what happened when SpannerIO was added.

It might have been covered by the google-cloud-platform scope. Java doesn't have spanner scopes, not sure if python had to reference them explicitly.

[kennknowles]

5b6a0ea#diff-21a78a52eca0c898070d58302127a9bb5cdb5de512ec16b9b3d945e0b84b694c

[lukecwik]

5b6a0ea#diff-21a78a52eca0c898070d58302127a9bb5cdb5de512ec16b9b3d945e0b84b694c only modifies Python's scopes and not Java's, I think @tvalentyn point still is valid.

[lukecwik]

If it is easy to do The Right Thing then go ahead and do it. But forcing arbitrary discipline on new contributors that we ourselves never complied with is not how we should run things.

We should be guiding contributors to produce the right solution, this isn't arbitrary.

[lukecwik]

#23644 is a proposed solution.

[ricardograca-scratch]

Thanks everyone! The final solution looks great, and I wouldn't have minded doing the right thing to help the project, but it would have taken me a long time since I'm not familiar with the project internals.

[lukecwik]

I'm happy that you're happy with the solution.

Hopefully we can spend more time with you on your next change.