"grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAUTHENTICATED" after leaving code running for multiple days #3212
Comments
|
This happened again (from pubsub_client.list_topics()). To make it clear, the auth works; it stops working after some time. |
|
Hmm, I am not aware of an upper limit on the lifetime of a client, but that does sound like the error you are getting. We will try to find out. |
lukesneeringer
added
priority: p2
|
Seeing the same thing on a script we left running on a standard GCP VM instance (Debian 8.7). Python 2.7.9. Some of the packages (prototype instance, pip freeze is quite long...): It runs great for some time, then we inevitably see this error if it's left running with nohup. Unfortunately I don't have timestamps to tell you roughly how long it takes to see it. |
|
Getting the same error here as well. The errors keeps coming for an interval of 5-10 minutes every hour. This is for a pubsub client that is running for a long time (hours). Here's a part of my |
|
@jonparrott Any ideas of how I can attack this problem? |
|
@lukesneeringer I'm gonna look into it. It seems that we need to enumerate and validate the failure modes for gRPC streaming when auth fails. |
|
@ensonic @ccmonnett @irri Can any of you show the complete code that invoking |
|
@lukesneeringer I think that gax just needs to retry grpc errors with |
|
@jonparrott Here's the simplified version of what I'm running: def create_pubsub_client():
client = pubsub.Client.from_service_account_json(PATH_TO_KEY, project=PROJECT_NAME)
return client
def receive_message(pubsub_client, topic_name, subscription_name):
topic = pubsub_client.topic(topic_name)
subscription = topic.subscription(subscription_name)
results = subscription.pull(return_immediately=True, max_messages=1)
for ack_id, message in results:
# do stuff
def main():
pubsub_client = create_pubsub_client()
while True:
try:
receive_message(pubsub_client, PUBSUB_TOPIC_NAME, PUBSUB_SUBSCRIPTION_NAME)
except Exception as e:
# log error
time.sleep(1)By moving the creation of the pubsub client into the while loop (and thus re-creating a new one every second) the problems go away. But I feel that is not a beautiful production-ready solution. |
|
I've got the same error message on GAE, but with pushing messages to pubsub. It usually appears every 30 minutes or so. The workaround is to create a new pubsub client before pushing a message. It is not ideal though, because it introduces an additional delay. |
|
@irri @mxmzdlv @ensonic can any of you try using a new credential type and seeing if this goes away? Make sure you have the latest google-auth and try this: import google.auth
import google.auth.jwt
credentials, project = google.auth.default()
jwt_credentials = google.auth.jwt.OnDemandCredentials.from_signing_credentials(credentials)
# Alternatively you can use OnDemandCredentials. from_service_account_file()
client = pub.Client(project=project, credentials=jwt_credentials) |
|
@jonparrott I've tried it and I still get the error message, but less frequently – always after 65 minutes and it seems to fix itself on subsequent 1-10 requests (the instance gets ~10 req/s). I've also tried changing |
|
@lukesneeringer at this point I have no idea. It seems somehow during the credentials refresh the old, expired token can still be used. Additionally, it doesn't appear that authentication errors are being retried appropriately. |
|
@mxmzdlv thanks for giving that a try. :) |
|
As another datapoint I can report that i've maxed out a 10k/month sentry.io free account with these errors from datastore.client.get_multi/put_multi and pubsub.subscription.acknowledge (the latter wrapped in |
|
@nathanielmanistaatgoogle is it possible we're doing something wrong with grpc authentication? iirc, you reviewed the code in google-auth for this. |
|
This should be fixed by https://github.com/GoogleCloudPlatform/google-auth-library-python/releases/tag/v1.0.1 which was just released. Try making sure that package is up-to-date and if this is still an issue, let us know. |
|
Does Because I think I bumped into the same issue. I'm using python2.7 and below is what my pip freeze shows: cachetools==2.0.0 |
|
@gylu 1.0.1 is the latest, but can you tell us more about the issue you're experiencing? This particular issue is that after ~60 minutes requests will fail with unauthenticated for ~5 minutes and then will work for ~60 minutes (repeat). |
|
@jonparrott My error was quite similar. I had it running for about an hour when this occurred (my code just stops, it doesn't try to keep on running, so I don't know about the ~5min-bad then ~60min-good cycle). I will try to rerun it and see if the same thing happens. Here's what my code is mainly doing. I am reading out of Pubsub and writing into a database.: But below is some of the printout I got when it failed: |
|
@gylu can you give me the following output from when you get this error? import datetime
print(pubsub_client.credentials)
print(pubsub_client.credentials.valid)
print(pubsub_client.credentials.expiry)
print(datetime.datetime.utcnow()) |
|
Same issue here. The code is running fine for a while, but suddenly I get this error: My code: |
|
@ppawiggers Note the different error codes: yours is Your issue seems more like #3888 than this one. |
|
@jonparrott @tseaver I am also experiencing the same issue while calling google vision APIs Here is the stacktrace:- And here is dump of my pip freeze, I am using google-auth version 1.1.1:- |
|
same here ... |
|
Update... |
|
Hello Everyone, I hope someone found a solution for this issue, I am still getting the error
I have installed the packages mentioned by @FingerLiu . But still I am getting the same error message. Note: Google SPeech API is enabled and I am using Python 3.6 Here is the code, thank you for your help. |
|
@sohamsil There are a couple of things you can try. First of all, we deprecated the gax library. Try upgrading the package by using Second, you can go to Please let us know if you need more assistance or this has solved your problem. |
|
@chemelnucfin Thanks for the prompt reply. I checked the speech API in But still getting the same error and also it gave me 403 request error.
Kindly look into this. |
Code is running on GKE. The docker image is based on ubuntu:xenial and has Python 3.5.2.
These are the packages:
When leaving the code running for several days we get a series of such exceptions:
later in the log we get:
The code where it starts to fails is a
subscription.pull(return_immediately=False, max_messages=10)wrapped into a @retrying.retry(stop_max_attempt_number=5).A similar looking error was reported in #2684 but this does not seem to be the same issue.
The text was updated successfully, but these errors were encountered: