AttributeError: 'SafeProtocol' object has no attribute 'session'

[ederenn]

Encountered on:

Linux
mainnet
build: https://buildbot.golem.network/buildbot/#/builders/2/builds/3670

appeared during regular computations

2019-02-06 15:13:23 ERROR    golem.client                        p2pservice.sync_network failed
Traceback (most recent call last):
  File "/home/buildbot-worker/worker/buildpackage_linux/build/golem/client.py", line 1404, in _run
  File "/home/buildbot-worker/worker/buildpackage_linux/build/golem/network/p2p/p2pservice.py", line 253, in sync_network
  File "/home/buildbot-worker/worker/buildpackage_linux/build/golem/network/p2p/p2pservice.py", line 902, in __send_get_peers
  File "/home/buildbot-worker/worker/buildpackage_linux/build/golem/network/p2p/peersession.py", line 171, in send_get_peers
  File "/home/buildbot-worker/worker/buildpackage_linux/build/golem/network/p2p/peersession.py", line 129, in send
  File "/home/buildbot-worker/worker/buildpackage_linux/build/golem/network/transport/session.py", line 201, in send
  File "/home/buildbot-worker/worker/buildpackage_linux/build/golem/network/transport/session.py", line 115, in send
  File "/home/buildbot-worker/worker/buildpackage_linux/build/golem/network/transport/tcpnetwork.py", line 289, in send_message
  File "/home/buildbot-worker/worker/buildpackage_linux/build/golem/network/transport/tcpnetwork.py", line 474, in _prepare_msg_to_send
AttributeError: 'SafeProtocol' object has no attribute 'session'

[etam]

Build from https://github.com/golemfactory/golem/pull/3843

[etam]

https://github.com/golemfactory/golem/blob/32bba1097cb3de9704b652c0c051894f923267c0/golem/network/transport/tcpnetwork.py#L464-L478

The exception happened at line 474, but there is a check at line 466. The only way this could happen is if another thread deletes session member. This can happen if connectionLost is called:

https://github.com/golemfactory/golem/blob/32bba1097cb3de9704b652c0c051894f923267c0/golem/network/transport/network.py#L110-L111

Sounds like #3660

[etam]

Another issue is that even if there was no race condition, if self.session is None would also raise AttributeError.

[kmazurek]

As discussed with @mfranciszkiewicz - without solving the actual underlying cause (#3660) a temporary fix would be to synchronise session via locking. Considering the cost of that vs the low frequency of this issue I'd say it's not worth patching in 0.19. @etam What do you think?

[etam]

I agree.

And for solving #3660 I think we should start with merging #3661 and then carefully start to look for possible bottlenecks and things that could be moved to separate threads. Because right now we just have unsynchronized threads mayhem.