Infinite loop if refresh token is invalid

[tiger154]

Hi there,

If I use an invalid refresh token, it goes into an infinite loop, not throwing an exception.
It keeps trying requesting itself again and again..

This is so frustrating as refresh token can be broken many reasons(Change password, etc)
If there are some options which can raise an exception, it would be much more easy to handle these situations.

Please let me know your opinion or guide me!

• Test Code

   @Test(expected = GoogleAdsException.class)
    public void infinite_loop_exception_test()  throws Exception{

        String personal_mcc_token = "1/SomeBrokenRefreshToken";
        String mcc_id = "xxxxxxx";
        String customer_client_id = "xxxxxxx";

        GoogleAdsClient googleAdsClient = testHelper.getGoogleClient(mcc_id,  personal_mcc_token);
        GoogleAdsServiceClient googleAdsServiceClient = googleAdsClient.getLatestVersion().createGoogleAdsServiceClient();
        SearchGoogleAdsRequest request = SearchGoogleAdsRequest.newBuilder()
                .setCustomerId(customer_client_id)
                .setPageSize(10000)
                .setQuery("SELECT campaign.id FROM campaign")
                .build();
        GoogleAdsServiceClient.SearchPagedResponse searchPagedResponse = googleAdsServiceClient.search(request);
    }

• Error logs

2019-08-13 16:56:20.178  WARN 14560 --- [          Gax-1] c.g.ads.googleads.lib.request.summary    : FAILURE REQUEST SUMMARY. Method: google.ads.googleads.v2.services.GoogleAdsService/Search, Endpoint: googleads.googleapis.com:443, CustomerID: 5262041796, RequestID: null, ResponseCode: UNAVAILABLE, Fault: Credentials failed to obtain metadata.
2019-08-13 16:56:20.180  INFO 14560 --- [          Gax-1] c.g.ads.googleads.lib.request.detail     : FAILURE REQUEST DETAIL.
Request
-------
MethodName: google.ads.googleads.v2.services.GoogleAdsService/Search
Endpoint: googleads.googleapis.com:443
Headers: {developer-token=REDACTED, login-customer-id=7987822632, x-goog-api-client=gl-java/1.8.0_161 gapic/ gax/1.45.0 grpc/1.21.0}
Body: customer_id: "5262041796"
query: "SELECT campaign.id FROM campaign"
page_size: 10000


Response
--------
Headers: null
Body: null
Failure message: null
Status: Status{code=UNAVAILABLE, description=Credentials failed to obtain metadata, cause=com.google.api.client.http.HttpResponseException: 400 Bad Request
{
  "error": "invalid_grant",
  "error_description": "Bad Request"
}
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1070)
	at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:193)
	at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:165)
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:151)
	at com.google.auth.Credentials.blockingGetToCallback(Credentials.java:113)
	at com.google.auth.Credentials$1.run(Credentials.java:99)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
}.
2019-08-13 16:56:22.746  WARN 14560 --- [          Gax-4] c.g.ads.googleads.lib.request.summary    : FAILURE REQUEST SUMMARY. Method: google.ads.googleads.v2.services.GoogleAdsService/Search, Endpoint: googleads.googleapis.com:443, CustomerID: 5262041796, RequestID: null, ResponseCode: UNAVAILABLE, Fault: Credentials failed to obtain metadata.
2019-08-13 16:56:22.747  INFO 14560 --- [          Gax-4] c.g.ads.googleads.lib.request.detail     : FAILURE REQUEST DETAIL.
Request
-------
MethodName: google.ads.googleads.v2.services.GoogleAdsService/Search
Endpoint: googleads.googleapis.com:443
Headers: {developer-token=REDACTED, login-customer-id=7987822632, x-goog-api-client=gl-java/1.8.0_161 gapic/ gax/1.45.0 grpc/1.21.0}
Body: customer_id: "5262041796"
query: "SELECT campaign.id FROM campaign"
page_size: 10000


Response
--------
Headers: null
Body: null
Failure message: null
Status: Status{code=UNAVAILABLE, description=Credentials failed to obtain metadata, cause=com.google.api.client.http.HttpResponseException: 400 Bad Request
{
  "error": "invalid_grant",
  "error_description": "Bad Request"
}
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1070)
	at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:193)
	at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:165)
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:151)
	at com.google.auth.Credentials.blockingGetToCallback(Credentials.java:113)
	at com.google.auth.Credentials$1.run(Credentials.java:99)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
}.
2019-08-13 16:56:23.128  WARN 14560 --- [          Gax-1] c.g.ads.googleads.lib.request.summary    : FAILURE REQUEST SUMMARY. Method: google.ads.googleads.v2.services.GoogleAdsService/Search, Endpoint: googleads.googleapis.com:443, CustomerID: 5262041796, RequestID: null, ResponseCode: UNAVAILABLE, Fault: Credentials failed to obtain metadata.
2019-08-13 16:56:23.129  INFO 14560 --- [          Gax-1] c.g.ads.googleads.lib.request.detail     : FAILURE REQUEST DETAIL.
Request
-------
MethodName: google.ads.googleads.v2.services.GoogleAdsService/Search
Endpoint: googleads.googleapis.com:443
Headers: {developer-token=REDACTED, login-customer-id=7987822632, x-goog-api-client=gl-java/1.8.0_161 gapic/ gax/1.45.0 grpc/1.21.0}
Body: customer_id: "5262041796"
query: "SELECT campaign.id FROM campaign"
page_size: 10000


Response
--------
Headers: null
Body: null
Failure message: null
Status: Status{code=UNAVAILABLE, description=Credentials failed to obtain metadata, cause=com.google.api.client.http.HttpResponseException: 400 Bad Request
{
  "error": "invalid_grant",
  "error_description": "Bad Request"
}
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1070)
	at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:193)
	at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:165)
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:151)
	at com.google.auth.Credentials.blockingGetToCallback(Credentials.java:113)
	at com.google.auth.Credentials$1.run(Credentials.java:99)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
}.
Disconnected from the target VM, address: '127.0.0.1:62310', transport: 'socket'
2019-08-13 16:56:26.095  INFO 14560 --- [       Thread-5] c.m.t.c.processor.AbstractJobProcessor   : Shutting down application...
2019-08-13 16:56:26.096  INFO 14560 --- [       Thread-5] c.m.t.c.processor.AbstractJobProcessor   : Shutdown complete.

Process finished with exit code 130

[nwbirnie]

@devchas please try to reproduce this issue. Check if the OAuth library allows us to detect an invalid refresh token. If so implement a check in the builder which rejects the refresh token with a new exception called LibrarySetupException which contains an enum of reasons.

[devchas]

Apologies for the delayed response here. I was able to reproduce the issue. Interestingly, the issue only exists for search requests. Mutate and searchStream request exhibit normal behavior. I've debugged fairly extensively and believe there is an issue with either the Gax or OAuth2 libraries because the call to GoogleAdsServiceClient.search ultimately creates a UnaryCallable instance, and that call method never returns.

I've opened an issue with the gax-java repo to see if they have any insight. I do think it's specific to the Java implementation of Gax because I tried to replicate the issue with the client library for Python, but it returned an error as expected without any infinite loop.

@nwbirnie any thoughts here?

[nwbirnie]

Pinged on the bug, I agree this looks like a bug in GAX. There is a workaround possible here which is to switch the RPC from paging -> streaming (i.e. call searchStream() instead of search()). We have upcoming changes to the code generation and I'm planning to get this resolved there. Please let us know if this is infeasible for you.

[nwbirnie]

Looks like we're tracking this in the upcoming microgenerator migration.

[debedb]

I have come here because I have seen this issue (with OAuth token) before; but also -- unsure if it is related -- I notice an issue with requests hanging when they result in an error sometimes. For instance, unspecified LoginCustomerId when it needed to be specified resulted in the program hanging. I, unfortunately, do not have a pithy case to reproduce at the moment but has there been any more information here?

[yshyya]

Hello,
I have a similar problem. However, using searchStream() couldn't be workaround.
I tried v10.1.0, v11.0.0.

// googleAdsServiceClient is created by invalid credentials

ServerStream<SearchGoogleAdsStreamResponse> stream =
        googleAdsServiceClient.searchStreamCallable().call(request);

// Never returns... 
stream.iterator().hasNext();

and I checked logs, the retries are repeating endlessly.

[nwbirnie]

I can confirm that this looks like a misbehavior of the retries. We appear to infinitely retry failed OAuth requests.

Logs here.

I'm guessing that we have the wrong settings for retries on OAuth errors. Will reach out internally to investigate and get back to you.

[Sayil8]

Hey guys,
it seems that you are already working on this issue, but I wanted to mention that the issue appears in with other methods too, not only search(). In our case it appears when using CampaignCriterionService/MutateCampaignCriteria method.
We build the GoogleAdsClient with the necessary properties and when the refresh token is not valid any more, we enter the infinite loop of requests.
If requested, I can add the logs and explain how to reproduce this.
Cheers

[nwbirnie]

Hey thanks for the ping. The underlying issue is being investigated here, I've added some more debugging info an reopened since I'm pretty sure that's the root cause.

[nwbirnie]

Yes, this is currently being worked on, was picked up a couple weeks ago. We should have an update here soon.

[ThalyVega]

#169

[danidaniel6462]

I am also waiting for a solution to this problem, I am pending for when they solve it

[wanqulou]

Can do this:

UserCredentials credentials = UserCredentials.newBuilder()
                .setClientId(clientId)
                .setClientSecret(clientSecret)
                .setRefreshToken(refreshToken)
                .build();

// if refresh token is invalid, throw exception
AccessToken accessToken = credentials.refreshAccessToken();

GoogleAdsClient.newBuilder()
                .setCredentials(credentials.toBuilder().setAccessToken(accessToken).build())
                .setDeveloperToken(developerToken)
                .setLoginCustomerId(loginCustomerId)
                .build();

[AnashOommen]

I like this as a temporary workaround until we get a fix.

…

On Tue, Aug 24, 2021, 11:10 PM Xiang Sun ***@***.***> wrote: Can do this: `UserCredentials credentials = UserCredentials.newBuilder() .setClientId(clientId) .setClientSecret(clientSecret) .setRefreshToken(refreshToken) .build(); AccessToken accessToken = credentials.refreshAccessToken(); return GoogleAdsClient.newBuilder() .setCredentials(credentials.toBuilder().setAccessToken(accessToken).build()) .setDeveloperToken(developerToken) .setLoginCustomerId(loginCustomerId) .build();` — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#169 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCOJCGKKNJSHBE6W3OD66LT6RNKZANCNFSM4ILIFPWQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[devchas]

Just updating this thread that the root cause seems to be caused by this issue. It looks like a design was in progress as of September 2021.

[jradcliff]

Current status is that the addition of the Retryable interface was released in google-auth-library 1.5.3, and that needs to be incorporated into grpc-java. That was attempted in grpc/grpc-java#9102, but then reverted in grpc/grpc-java#9118 due to other Google dependencies. We're waiting for those issues to be resolved.

Internal ref: b/231441733

[Still4]

you can manually shutdown after certain time

try (GoogleAdsServiceClient client = context.getLatestVersion().createGoogleAdsServiceClient()) {
    final AtomicBoolean finish = new AtomicBoolean(false);
    ResponseObserver<SearchGoogleAdsStreamResponse> responseObserver;
    responseObserver = new ResponseObserver<SearchGoogleAdsStreamResponse>() {
        @Override
        public void onStart(StreamController controller) {
        }

        @Override
        public void onResponse(SearchGoogleAdsStreamResponse response) {
            //your code after getting correct response
        }

        @Override
        public void onError(Throwable t) {
            //you may export log here
            finish.set(true);
        }

        @Override
        public void onComplete() {
            finish.set(true);
        }
    };
    client.searchStreamCallable().call(apiRequest, responseObserver);
    long callStart = System.currentTimeMillis();
    //wait stream call finish
    long expire = 15000;
    while (!finish.get()) {
        long callCheck = System.currentTimeMillis();
        if (callCheck - callStart > 15000) {
            client.shutdownNow();
            //you may add shutdown log
            break;
        }
    }
} catch (Exception e) {
}

[weichangdong]

I had the same situation, cancel the multiple threads didn't work。

 log.info("accountInfoMonitor-start-1");
        FutureTask<List<CustomerClient>> futureTask = new FutureTask<>(new Callable<List<CustomerClient>>() {
            @Override
            public List<CustomerClient> call() {
                return googleCustomer.getCustomerInfo(appConfig);
            }
        });
        ExecutorService executor = Executors.newSingleThreadExecutor();
        executor.submit(futureTask);


        List<CustomerClient> list = new ArrayList<>();
        log.info("accountInfoMonitor-start-2");
        try {
            list = futureTask.get(120, TimeUnit.SECONDS);
        } catch (InterruptedException e) {
            futureTask.cancel(true);
            log.error("InterruptedException msg:{}", e.getMessage());
        } catch (ExecutionException e) {
            futureTask.cancel(true);
            log.error("ExecutionException msg:{}", e.getMessage());
        } catch (TimeoutException e) {
            futureTask.cancel(true);
            log.error("TimeoutException msg:{}", e.getMessage());
        }
        log.info("accountInfoMonitor-start-3");

[jradcliff]

Have you tried explicitly calling refreshAccessToken() as mentioned in a previous comment? That will fail immediately if you have an invalid refresh token.

Can do this:

UserCredentials credentials = UserCredentials.newBuilder()
                .setClientId(clientId)
                .setClientSecret(clientSecret)
                .setRefreshToken(refreshToken)
                .build();

// if refresh token is invalid, throw exception
AccessToken accessToken = credentials.refreshAccessToken();

GoogleAdsClient.newBuilder()
                .setCredentials(credentials.toBuilder().setAccessToken(accessToken).build())
                .setDeveloperToken(developerToken)
                .setLoginCustomerId(loginCustomerId)
                .build();

[weichangdong]

Thank you for your answer. I have not tried this method, my problem is because I deliberately modified the token to be wrong, because the token is not permanent, so I want to test this situation.
This refresh token is used elsewhere, I am afraid that calling this function will really change the refresh token, which will affect the use of other places.

Now that I've solved it, I still have to use google client's own shutdown.

[chiccomask]

Still no fix for this?

[jradcliff]

This needs a fix in grpc-java for GRPC will keep retrying (until RPC timeout) if a user uses a bad service account key · Issue #6808 · grpc/grpc-java.

A suboptimal workaround would be to remove UNAVAILABLE from the list of retryable codes. For example, for a GoogleAdsService.search call, instead of:

    googleAdsServiceClient.search(request);

you could do:

    googleAdsServiceClient
          .searchCallable()
          .call(
              request,
              GrpcCallContext.createDefault()
                  // Overrides the list of retryable codes to exclude 'UNAVAILABLE'.
                  .withRetryableCodes(ImmutableSet.of(Code.DEADLINE_EXCEEDED)));

However, this is suboptimal because there will be occasional UNAVAILABLE errors that should be retried but are not due to invalid OAuth credentials. By removing UNAVAILABLE, the library will not retry those other types of requests.

[jacob-tolar-bridg]

That grpc-java change looks like it is in 1.63 which was pushed to maven central Apr 4.

Edit: I tested it and found no change in behavior. In fact, although grpc/grpc-java#6808 was marked closed for the 1.63 milestone, the commit was reverted (but only on the v1.63.x branch, it's still in master). So, one of the workarounds described above is still required.

[jradcliff]

@jacob-tolar-bridg , thanks for highlighting and for testing. I'm following up with my gRPC contacts on this.

-Josh, Google Ads API Team

[jradcliff]

We align the google-ads-java dependencies with those of the libraries-bom project for interoperability between this library and other Google Cloud libraries. The latest release of the libraries-bom project depends on grpc 1.62.2, but I'm checking periodically for updates.

Thanks,
Josh

[psrivardhan]

Finally it is fixed. Updating grpc-java dependencies to 1.64.0 worked for me
implementation 'io.grpc:grpc-protobuf:1.64.0' implementation 'io.grpc:grpc-stub:1.64.0' implementation 'io.grpc:grpc-xds:1.64.0'

See here: grpc/grpc-java#6808 (comment)

[jradcliff]

Thanks for confirming. I'm still waiting for libraries-bom to pick up this change, per my previous comment.

[pranjal-jadhav-invimatic]

@jradcliff
Is there any ETA to release the updated library

[jradcliff]

Good news! Version 33.0.0, released earlier today, includes an upgrade to gRPC 1.65.1 where this issue has been fixed.