OAuth Access Token Refresh Does Not Release If Account Has Been Deleted

[PeterLavetsky]

com.google.api-ads:google-ads:17.0.1

searchStreamCallable calls for customer_ids that have been deleted do not release from the doRefreshAccessToken call in a timely manner.

This call doesn't generate a request-id so I can't provide it.

Request
-------
MethodName: google.ads.googleads.v10.services.GoogleAdsService/SearchStream
Endpoint: googleads.googleapis.com:443
Headers: {developer-token=REDACTED, login-customer-id=ABC, x-goog-api-client=gl-java/11.0.13 gccl/17.0.1 gapic/17.0.1 gax/2.6.1 grpc/1.41.0}
Body: customer_id: "XYZ"
query: "select customer.auto_tagging_enabled, customer.call_reporting_setting.call_conversion_action, customer.call_reporting_setting.call_conversion_reporting_enabled, customer.call_reporting_setting.call_reporting_enabled, customer.conversion_tracking_setting.accepted_customer_data_terms, customer.conversion_tracking_setting.conversion_tracking_id, customer.conversion_tracking_setting.conversion_tracking_status, customer.conversion_tracking_setting.cross_account_conversion_tracking_id, customer.conversion_tracking_setting.enhanced_conversions_for_leads_enabled, customer.currency_code, customer.descriptive_name, customer.final_url_suffix, customer.has_partners_badge, customer.id, customer.manager, customer.optimization_score, customer.optimization_score_weight, customer.pay_per_conversion_eligibility_failure_reasons, customer.remarketing_setting.google_global_site_tag, customer.resource_name, customer.status, customer.test_account, customer.time_zone, customer.tracking_url_template, metrics.optimization_score_uplift, metrics.optimization_score_url from customer "

Response
--------
Headers: null
Body: null
Failure message: null
Status: Status{code=UNAVAILABLE, description=Credentials failed to obtain metadata, cause=com.google.api.client.http.HttpResponseException: 400 Bad Request
POST https://oauth2.googleapis.com/token
{ 
  "error": "invalid_grant",
  "error_description": "Account has been deleted"
}
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1116)
        at com.google.auth.oauth2.UserCredentials.doRefreshAccessToken(UserCredentials.java:273)
        at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:190)
        at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:257)
        at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:254)
        at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
}.

This request iterates over and over again, logging the error fast but not throwing any exception to move on from.

The following is from a single request for a bad customer_id:

2022-02-21 11:53:26,894 INFO  RequestLogger - FAILURE REQUEST DETAIL.
2022-02-21 11:53:29,004 INFO  RequestLogger - FAILURE REQUEST DETAIL.
2022-02-21 11:53:31,269 INFO  RequestLogger - FAILURE REQUEST DETAIL.
2022-02-21 11:53:38,543 INFO  RequestLogger - FAILURE REQUEST DETAIL.
2022-02-21 11:53:43,506 INFO  RequestLogger - FAILURE REQUEST DETAIL.
2022-02-21 11:53:52,240 INFO  RequestLogger - FAILURE REQUEST DETAIL.
2022-02-21 11:53:52,427 INFO  RequestLogger - FAILURE REQUEST DETAIL.
2022-02-21 11:54:10,388 INFO  RequestLogger - FAILURE REQUEST DETAIL.
2022-02-21 11:54:39,543 INFO  RequestLogger - FAILURE REQUEST DETAIL.
2022-02-21 11:54:47,258 INFO  RequestLogger - FAILURE REQUEST DETAIL.
2022-02-21 11:55:37,705 INFO  RequestLogger - FAILURE REQUEST DETAIL.
2022-02-21 11:55:44,776 INFO  RequestLogger - FAILURE REQUEST DETAIL.
2022-02-21 11:55:58,925 INFO  RequestLogger - FAILURE REQUEST DETAIL.

We just moved to 17.0.1 but I don't think this behavior was introduced in 17+

Pete

[schabe77]

this looks like a duplicate to #169

[jradcliff]

@schabe77 , you are correct. In both cases, the gax library we depend on gets a 400 response and retries.

@PeterLavetsky , you don't get a request-id for that one because the failure occurs when the library hits the OAuth endpoint, not the Google Ads API endpoint, and the former does not return a request-id.

With v10, you may be able to minimize this issue by fetching the new status fields before attempting to make a request for a given customer. If you are working with test accounts, please note that they'll have status = CLOSED, but customer_client.test_account = true.

Finally, the workaround mentioned in #169 may be helpful.

[jradcliff]

Closing as duplicate of #169.

[PeterLavetsky]

Thanks @jradcliff ... we've implemented this manual access token refresh strategy.