Datastore timeout when internet drops intermittently and app in background for too long

[nubpro]

Describe the bug
Datastore loses realtime sync as soon as network disconnects and reconnect immediately. Few minutes later, Datastore shows up a bunch of warning errors.
15:31:20.149 client_log [WARN] 31:20.155 DataStore - Timeout disconnect

To Reproduce
Steps to reproduce the behavior:

1. Open the React Native app on device A and B.
2. Starts the syncing process with Datastore.start(). (Client is connected with Server via Websocket)
3. Unplug the LAN cable from your router, reconnect it immediately.
4. Insert new record on device B with an active internet connection (different network).
5. Device A does not receive the record from Device B.
6. On device A, [WARN] 31:20.155 DataStore - Timeout disconnect shows up few minutes later.

Environment
iPad (iOS 13)
React Native

Amplify version
├─ @aws-amplify/analytics@3.2.1
├─ @aws-amplify/api-graphql@1.1.0
├─ @aws-amplify/api-rest@1.1.0
├─ @aws-amplify/api@3.1.17
├─ @aws-amplify/auth@3.2.14
├─ @aws-amplify/cache@3.1.17
├─ @aws-amplify/core@3.4.0
├─ @aws-amplify/datastore@2.2.4
├─ @aws-amplify/interactions@3.1.17
├─ @aws-amplify/predictions@3.1.17
├─ @aws-amplify/pubsub@3.0.18
├─ @aws-amplify/storage@3.2.7
├─ @aws-amplify/ui@2.0.2
├─ @aws-amplify/xr@2.1.17
├─ aws-amplify-react-native@4.2.1
└─ aws-amplify@3.0.18

Additional context
Based on my findings, NetInfo checks whether the internet is reachable every 60 seconds.
If the device is disconnected and reconnected to the Internet before NetInfo performs the check, AppSync & Datastore do not have any idea there was a connection drop. Therefore, AppSync does not re-establish the websocket connection even though it's already broken.

My current workaround
I have lowered down both reachabilityLongTimeout and reachabilityRequestTimeout to 5 seconds.
With this change, NetInfo is able to tell whether the WiFi has an active internet connection or not much quicker.
It kinda fixes this issue as Appsync can re-establish the Websocket connection much more responsively.

[elorzafe]

@nubpro You see the warning but dont see the data?

[nubpro]

@nubpro You see the warning but dont see the data?

No data is received by the client, as stated in step 5.

[mdoesburg]

I have also witnessed this timeout disconnect warning after closing my laptop while my app was still running on iOS Simulator. The only way to make the app/connectivity work again was a reload of the simulator or reboot of the app.

[nubpro]

The main problem is that in our productiom environment, we could not tell based on just looking at the app. We would only know when no new records are getting received by the device and this is not great, imagine missing out all the orders placed by your customers. We are losing money here and it impacts the business.

Is there a way that we could monitor the healthiness of the websocket connection between the client and the server? If so, we could at least restart the app and reinitiate the websocket. Because at the moment, there is no indicator from datastore and we could not catch the datastore timeout error programatically.

This is huge to us, please provide a temporary solution to us.

[nubpro]

I would also like to add, if the app is in background mode for too long, the subscription will fail - Datastore - Timeout Disconnect.

Listening to the Datastore's networkStatus from Hub listener, is not useful in detecting a socket failure. It still shows active network despite the timeout disconnection.

Diving into the source code, the exact warning is thrown at line 360 @ subscription.ts.

[nubpro]

Can we at least get an acknowledgement from DataStore engineers on this?
@sammartinez

[sammartinez]

Yes, @nubpro I can let them know. Apologizes on the delay!

[iartemiev]

Thank you for the detailed reproduction instructions, @nubpro. I was able to reproduce. Took exactly 5 minutes for device A to show the error message you specified in step 6. We will start investigating and working on a fix for this.

[nubpro]

Just wondering is there an update to this? @iartemiev

My temporary workaround is to dispatch a message when the subscription throw `Timeout disconnect` error, if it does, I'll clear and start datastore again. ``` Hub.dispatch('datastore', { event: 'subscriptionProcessorError', data: { message }, }); ```

if (data.message === 'Timeout disconnect') {
 DataStore.clear()
      .then(() => {
        return DataStore.start();
      })
     .catch(() => {});
}
</s>

[nubpro]

By lowering reachabilityLongTimeout significantly at NetInfo configuration, NetInfo will have a better chance knowing that there's a connection drop.

However, there is still a tendency where the subscription has failed but DataStore didn't attempt to restart.
Hence, I've submitted a PR(#6366) that fixes the built-in disconnection handler. I'm not sure why it wasn't called in the first place.

--
On another note, 5 minutes seems to be too long for websocket health check.
How could I lower connectionTimeoutMs in AppSync?

[nubpro]

The current implementation does not allow subscriptions to reinitiate when PubSub returns Subscription timeout {query: ..., variables: ...} which is caused by poor network condition when initiating the subscription the first time.

Is it possible that we could do reconnection in this case?

[amhinson]

This issue appears to be related to #6488. Since the connected PR #6366 for this issue has been merged, I'm going to close this so any further discussion happens on the related issue.

[github-actions]

This issue has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs.

Looking for a help forum? We recommend joining the Amplify Community Discord server *-help channels or Discussions for those types of questions.