AbortionError(code=StatusCode.UNAUTHENTICATED, details="Request had invalid authentication credentials.")

[forsberg]

Very seldom, like 9 out of 450k publish requests, we get the following error:

AbortionError(code=StatusCode.UNAUTHENTICATED, details="Request had invalid authentication credentials.")

Retrying 2 times doesn't help, and we loose the message.

What's the recommended way of dealing with this problem? Retry until it works? Should application code have to deal with it, or is it something that should be handled automatically by SDK?

Versions, traceback etc is the same as in #2683.

[daspecster]

@forsberg thanks for reporting!

In #2683, you say

Retrying this seem to always succeed.

but here you're saying that retrying doesn't always work?

There has been talk of exposing some of the retry methods that we use in our system tests but that hasn't completely been decided yet. It does sound like having an option for the library to handle retrying is the direction we would like to go.

There are some good python retry libraries out there that will use exponential backoff to retry the request.

[dhermes]

@forsberg AbortionError indicates you are using an old version of this library. That was a beta feature of grpc and we have been using the stable version for some time.

Can you explain how you are installing?

[dhermes]

@forsberg FWIW we've seen something similar in #1917. The issue is essentially one of timing: your token may not be expired when we check the expiry before making a request, but by the time it reaches the server, it is expired. Do you know what kind of credentials you are using? (e.g. service account or a user credentials)

[forsberg]

We have a very simple retry logic here - it will simply try 3 times in a row. For the problem in #2683, this seems to have done the trick. For this, it doesn't. I guess it's matter of timing.

I'll make sure we do a fresh install from master without old crap lying around on Monday to see if problem still appears.

Agree that having built-in retry functionality would be very convenient. Or at least have documented what kind of exceptions are worth retrying.

[forsberg]

And to answer your question: It's a service account.

[forsberg]

After fixing our borked install to use a relatively fresh master e1fbb6bc, we have not observed this particular error anymore, but we have not had it running for a very long time. I'll report back again when we've had it running for a week or so without problems.

[dhermes]

@forsberg Thanks for the update. This error is in some sense unavoidable because we check "is the token valid" (and here) before sending, but by the time it's sent it may have become invalid. (However, I'm fairly certain the server gives 5 minutes of wiggle room, which isn't part of the expiry so it seems "impossible".)

[lukesneeringer]

Hello! :-)
First, a mea culpa; I have not done as good of a job at keeping up with issues as I should have. If you are getting this (I admit it) cut and paste, it is likely because your issue sat for too long.

In this case, I have been in the process of making a radical update to the PubSub library (#3637) to add significant performance improvements and a new surface, which we hope to launch soon. As such, I am clearing out issues on the old library. It is my sincere goal to do a better job of being on top of issues in the future.

As the preceding paragraph implies, I am closing this issue. If the revamped library does not solve your issue, however, please feel free to reopen.

Thanks!