HTTP2Transport race may lead to deadlock

[jshslt]

Hi,
I think there's a race in HTTP2Transport that can lead to deadlock if the networkLoop thread tries to signal out a successful connection, while another thread has asked for the network thread to terminate (and is waiting to join).

full scenario -
A - if any client thread calls MessageRouter::disable. MessageRouter lock is taken. MessageRouter::disconnectAllTransportsLocked is called. the transports are iterated, HTTP2Transport::disconnect is called, which calls join() on the SDK network thread.

B - SDK network thread, when connection is just established, HTTP2Transport::networkLoop calls the onConnected callback MessageRouter::onConnected, which tries to take the MessageRouter lock.

if B happens while A is in process, deadlock.

please advise?

[sanjayrd]

Thanks for pointing this out @jshslt. We haven't had a chance to dive deep into this yet, but we will definitely investigate the matter to see what we can do.

In the meantime, could you expand on the use case of this? More specifically, is this an issue that you actually foresee happening in your production code or just something you noticed as you were going through the code?

Thanks!
Sanjay

[jshslt]

@sanjayrd we happened to hit it for real. We've been getting a lot of SERVER_SIDE_DISCONNECTs, and sometimes the SDK appears to get in a PENDING->CONNECTED->PENDING->CONNECTED loop because of these - when we catch this starting to happen, we sometimes forcibly and fully disconnect the transport with MessageRouter::disconnect(), to rest a bit before trying to reconnect again.

[garmin-coleman]

@jshslt - FYI we were also seeing the PENDING->CONNECTED loop you mention. For us it would reliably happen if we accidentally had two clients using the same refresh token. The two clients would essentially ping-pong, causing the server to kick the other off when reconnecting.

[JamieMeyers]

We have confirmed the problem and are working on a fix.

Thanks,
Jamie

[scotthea-amazon]

A fix for this has been pushed in version 0.4.1.