-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PubSub: set_exception can only be called once.
can still occur "erroneously"
#4463
Comments
I also got this if I try to listen on a subscription that doesn't exist, FYI. Listening for messages on projects/python-docs-samples-tests/subscriptions/my-subs
Exception in thread Consumer helper: consume bidirectional stream:
Traceback (most recent call last):
File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 51, in error_remapped_callable
return callable_(*args, **kwargs)
File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/grpc/_channel.py", line 342, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.NOT_FOUND, Resource not found (resource=my-subs).)>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_consumer.py", line 235, in _blocking_consume
for response in response_generator:
File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/grpc/_channel.py", line 348, in __next__
return self._next()
File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 53, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.NotFound: 404 Resource not found (resource=my-subs).
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/jonwayne/.pyenv/versions/3.6.0/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/Users/jonwayne/.pyenv/versions/3.6.0/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_consumer.py", line 249, in _blocking_consume
self._policy.on_exception(exc)
File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/policy/thread.py", line 162, in on_exception
self._future.set_exception(exception)
File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/google/cloud/pubsub_v1/futures.py", line 159, in set_exception
raise RuntimeError('set_exception can only be called once.')
RuntimeError: set_exception can only be called once. |
Good data, thanks Jon! |
@jonparrott your example will trigger two |
set_exception can only be called once.
can still occur "erroneously"set_exception can only be called once.
can still occur "erroneously"
I think if there's any none retry-able exception occurred, the helper loop should just exit immediately by calling stop_consume. |
@shileiyu I totally agree with you. I am currently wrestling this issue. The problem is that if it calls |
@dhermes BTW: I don't think the KeyboardInterrupt would ever happen on the helper thread, not only because only main thread would receive the system signals, but also if KeyboardInterrupt ever occurred before, the thread should stuck as well. |
We can effectively do this by setting
I haven't been able to trigger the |
OK I am going to cut |
OK https://pypi.org/project/google-cloud-pubsub/0.29.2/ is officially released. |
For example, If I add
just before
future.set_exception()
, then, when forcing a failure by:I see (focus on 00117233 and 00185066):
This is PROGRESS! It shows
67.833
seconds after the firstUNAVAILABLE
, a second one occurs and also tries to callset_exception()
.This reveals the real bug: the subscriber doesn't exit after the first call to
set_exception()
.UNAVAILABLE
agrees with what we've previously seen. That is, after ~60 seconds of inactivity, a gRPC client will raise a localUNAVAILABLE
(i.e. no round trip to the backend) and then it will become magically usable immediately after raising.The text was updated successfully, but these errors were encountered: