Error 401 after some time running

[gfaramaz]

Hello,

I use the same BigQuery object to execute my queries and about once in a day i get this error

Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

The problem has already been reported here googleapis/google-cloud-node#2139 and then has been transferred and fixed as a google-auth-library-nodejs issue (googleapis/google-auth-library-nodejs#133).

I am sure that the problem has been fixed in google-auth-library-nodejs but @google-cloud/bigquery does not rely on this package and still have the issue.

Environment details

• OS: Debian 8
• Node.js version: 10.9.0
• npm version: 6.2.0
• @google-cloud/bigquery version: 4.1.0

Steps to reproduce

1. Use a singleton BigQuery object created with service account credentials
   new BigQuery({ credentials: { client_email: 'foo@bar.com', private_key: 'xyz' } })
2. Keep making requests using that object.

[callmehiphop]

@gfaramaz BigQuery does use google-auth-library-nodejs however, it does so via @google-cloud/common package. Given how old the mentioned issues are, I think it's safe to assume that fix has found its way here.

Couple of followup questions

• Did you test on an older version of the client?
• Do you have an approximate guess on how much time passes (or how many requests go off) until the errors start?

@JustinBeckwith any chance you've been seeing similar errors anywhere else?

[gfaramaz]

@callmehiphop We used version 2.0.6, and tried to update to 4.1.0, the error is still the same.

Here is a log of the error for the lasts 7 days:

June 3rd 2019, 04:17:04.047
June 2nd 2019, 22:16:44.173
June 1st 2019, 22:16:06.027
May 30th 2019, 10:15:22.021
May 29th 2019, 22:15:01.034

[JustinBeckwith]

👋 We haven't seen widespread reports of this, no. May I ask - which runtime environment is this in? Cloud Functions? Kubernetes Engine? App Engine?

[gfaramaz]

@JustinBeckwith We are running 3 pods of this particular code in kubernetes. Here is the last details I have:
For one pod only:

• Started June 4th 2019, 16:03:41
• 1st error June 4th 2019, 19:03:47
• 2nd error June 5th 2019, 07:03:58
• 3rd error June 5th 2019, 10:04:00

[callmehiphop]

@gfaramaz are you still experiencing this issue? Looks like the auth library has since made a patch and I'm wondering if it resolved this.

[gfaramaz]

Hello @callmehiphop , unfortunately I couldn't say, in order to solve this problem I have wrapped my BigQuery client objet into a singleton with a few hours expiration time, works fine.

[aaleksandrov]

I also experience (presumably) the same issue.
After some time running requests to BQ fail with Bearer realm="https://accounts.google.com/", error="invalid_token"
P.S. Using google-auth-library-nodejs 5.1.2 , @google-cloud/bigquery version: 4.1.8
P.P.S. I debuged a bit and found that google-auth-library tries to refresh the access token but google auth api keeps returning the same access token until it expires. So maybe on nodejs-bigquery (or maybe it should be google-cloud/common? ) side you can implement logic to refresh access token if request to BQ fails with 401?

[JustinBeckwith]

Greetings folks! I am 84% sure we fixed this one :) There was a problem in previous versions of google-auth-library that was preventing token refreshes from happening when you'd expect them to. The fix was here:
googleapis/google-auth-library-nodejs@fecd4f4

If you are still experiencing this - please let me know! We would be happy to re-open the bug if it's still a problem.