Appsync graphQL connection closed error

[mchitson]

Hi I am using subscription via graphQL and getting frequent connection closed error from Appsync, for no apparent reason. It seems to happen at random times. Don't know how to reproduce it.

My .js code:

subscribeExhibitorNotifications = async() => {
try {
this.exhitorStatusSubscription = await API.graphql(graphqlOperation(onCreateOveceventsExhibitorStatus));
if (this.exhitorStatusSubscription) {
this.exhitorStatusSubscription.subscribe({
next: (event) => { ...
}
else {
....
}
},
error: error => {
console.log("error in graphQL onCreateOveceventsExhibitorStatus: ", error);
}
});
}
}
catch (e) {
console.log("error in graphQL onCreateOveceventsExhibitorStatus: ", e)
}
}

Schema:

export const onCreateOveceventsExhibitorStatus = /* GraphQL */ subscription OnCreateOveceventsExhibitorStatus { onCreateOveceventsExhibitorStatus { id datetime status } };

export const createOveceventsExhibitorStatus = /* GraphQL */ mutation CreateOveceventsExhibitorStatus($id: String!, $status: String!) { createOveceventsExhibitorStatus(id: $id, status: $status) { id datetime status } };

[mchitson]

for more info, i also see a comment from webpack after the appsync error:

Not sure if it's react closing the connection or appsync causing react to print out that command.

that's my version of react:

"react": "^16.13.1",
"react-dom": "^16.13.1",
"react-scripts": "3.4.3"

[SalvoCozzubo]

Same here. I have a react app with three subscriptions and all subscriptions received a "Connection Closed" in a complete random way.

Sometimes the connections are stable and sometimes they are closed after a couple minutes.
I will continue to investigate.

[mchitson]

Hello amplify guys, any luck?

[chunyanj]

Hi, I’m from the AppSync team. I’m happy to look into the issue you’ve been seeing, would anyone impacted be able to share a rough timestamp (open time and drop time), api id, and region for a connection that dropped? Sorry for the issues!

[mchitson]

Hi Chunyanj is it possible to send you a private message regards to API id?
In the meanwhile the other info:

region is us-east-1
open time probably around 10am EST (not quite sure if i am remembering the open time accurately), drop time around 1:35pm EST, all 29th oct 2020.

[chunyanj]

Thank you for the response! We have started to investigate. The API id has been recorded and you could delete the comment if you want. We will get back to you when we have update.

[mchitson]

great - thank you.

[mchitson]

hi Chunyan

I have another question and i have been seeing those questions over the web but no response - if a device is online and currently get subscriptions, then goes offline and then online again, will the subscription still work and then get the latest mutation that was missed?

[mchitson]

hi there -> any upates? thanks!

[chunyanj]

Hey mchitson, we are actively investigating the ticket. Sorry for any inconvenience. Please expect an update today.

[chunyanj]

As for the question above, unfortunately the answer is no. Typically, people could achieve this using delta sync. The doc for delta sync is here: https://docs.aws.amazon.com/appsync/latest/devguide/tutorial-delta-sync.html

[mchitson]

thanks but the document is not very detailed. it's unclear to me whether i need to use datastore with it or what datasync client to use. datastore also does not seem to recover subscriptions - at least not the way i see it documented.

[chunyanj]

Hey mchitson, we have root caused it.

Sorry for the issue. We dove deep to confirm, and this is an issue we’ve seen reported a few times where code deployments on our end can result in connections temporarily being broken up. We’re investigating a few avenues to alleviate some of these headaches, both on the client end and on our end. Thanks!

[chunyanj]

Thanks for you feedback of the doc, we'll put this in our backlog and discuss improving it!

[mchitson]

Thank you Chunyan So going forward there shouldn’t be a connection closed issue?

…

On Nov 3, 2020, at 8:24 PM, chunyanj ***@***.***> wrote:  Hey mchitson, we have root caused it. Sorry for the issue. We dove deep to confirm, and this is an issue we’ve seen reported a few times where code deployments on our end can result in connections temporarily being broken up. We’re investigating a few avenues to alleviate some of these headaches, both on the client end and on our end. Thanks! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[mchitson]

Also I am trying Datastore as I would like my subscriptions to be up to date when a device is temporarily disconnected and reconnected. I am however getting those errors: DataStore - subscriptionError Connection failed: {"errors":[{"message":"Validation error of type FieldUndefined: My package.json: "dependencies": { "@aws-amplify/cli": "^4.32.0", "@aws-amplify/datastore": "^2.7.2", "@testing-library/jest-dom": "^5.11.4", "@testing-library/react": "^11.1.0", "@testing-library/user-event": "^12.1.10", "aws-amplify": "^3.3.7", "react": "^17.0.1", "react-dom": "^17.0.1", "react-scripts": "4.0.0", "web-vitals": "^0.2.4" }, Secondly, am also seeing a timeout disconnect on appSync at less than 5mins. Is there a way to re-subscribe with appsync after this?

…

On Nov 3, 2020, at 8:24 PM, chunyanj ***@***.***> wrote: Hey mchitson, we have root caused it. Sorry for the issue. We dove deep to confirm, and this is an issue we’ve seen reported a few times where code deployments on our end can result in connections temporarily being broken up. We’re investigating a few avenues to alleviate some of these headaches, both on the client end and on our end. Thanks! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7070 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALSSPFRR6K7C2TBXDVKGYNLSOCUNZANCNFSM4TBP7NZA>.

[mchitson]

Hello Chunyan

The connection closed yet again last night around: 10:41 pm EST. I couldn't quite figure out from your response whether that problem has been solved or is still being solved after the root cause was identified. Thanks!

[mchitson]

Hello

I have yet another connection close at 5:15Pm 11/4/2020, i am going live soon and this can not happen so frequently, is this being fixed now that you have identified the root cause?

[chunyanj]

Hey mchitson,

Sorry for the ambiguity we made! The issue hasn't been solved 100% but we have taken actions to reduce the possibility of happening. We believe we can keep the frequency of happening in a very low rate.

As for the re-subscribe question, we'll provide an update later.

Thanks for the feedbacks!

[mchitson]

Hey Chunyan Am not sure if it’s as a result of what you guys have done to improve things, but I am getting disconnected even more often now! Like literally 20 mins ago!

…

On Nov 4, 2020, at 8:06 PM, chunyanj ***@***.***> wrote: Hey mchitson, Sorry for the ambiguity we made! The issue hasn't been solved 100% but we have taken actions to reduce the possibility of happening. We believe we can keep the frequency of happening in a very low rate. As for the re-subscribe question, we'll provide an update later. Thanks for the feedbacks! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7070 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALSSPFXGHTBLY2QKPFOGT23SOH275ANCNFSM4TBP7NZA>.

[mchitson]

i am still getting that connection closed issue happened yesterday again, barely couple of hours after my app started.

11/9/2020 10:19:12 PM Objecterror: errors: Array(1)0: {message: "Connection closed"}length: 1__proto__: Array(0)proto: Objectprovider: AWSAppSyncRealTimeProvider {_config: {…}, socketStatus: 0, keepAliveTimeout: 300000, subscriptionObserverMap: Map(0), promiseArray: Array(0), …}proto: Object
GraphQLServices.js:79

[chunyanj]

Hey mchitson,

Thanks for the feedback! We'll investigate it again to see if it is the same root cause.

[abdulqudoos]

I also faced connection closed issue today.

[abdulqudoos]

{
"provider": {

"socketStatus": 0,
"keepAliveTimeout": 300000,
"subscriptionObserverMap": {},
"promiseArray": [],
"awsRealTimeSocket": null,
"keepAliveTimeoutId": 202

},
"error": {
"errors": [
{
"message": "Connection closed"
}
]
}
}

[EliasCoelho]

Good afternoon, this uncaugth exception was raised in console for my web apps.
This exception can reflect an security breach for all api environment that it working with graphql...

Steps:

1. activate airplane mode
2. desactivate airplane mode
3. move to other tabs in browser
4. go the tabs with the web app and this error show for me in the console

[Mootook]

This is happening in my team's application, as well.

We use appsync graphql and vue apollo composable, and I see this error quite a bit.
From a bit of trial/error, the issue seems to arise from subscribing/unsubscribing in quick succession.

Vue apollo composable handles subscriptions internally in accordance with component lifecycles (from my understanding), so in this case, mounting and unmounting a component that subscribes/unsubscribe quickly will produce this error.
If I wait some time in between mounting/unmounting, the error is less common.
Not sure if that's actually the root cause or a way to get consistent behavior, just wanted to share in case it could help.

[giricr]

@Mootook i doubt subscribing/unsubscribing in quick succession is the cause as i see this issue where i do not even unsubscribe the listeners. If i leave the tab open for some time i see the connection closed error after approximately(not sure) 15 min .

[mikhail-shekhtman]

hey @mchitson, @chunyanj was the issue eventually resolved? Were you using MQTT as transport or WebSocket?

Can we experience the same as what was fixed at MQTT year ago and now connection is closing on WebSocket transport?

[michaeljajou]

Any updates to this issue?

[IgorDePaula]

I want put a digital store using amplify, now I think that is not secure with all this problems and almost 700 issues.

[david-mcafee]

Hey @mitchitson! After digging into this further, it looks like there are a few different root causes to this issue, and from what I can tell, none appear to be a problem with Amplify itself. Here's what I've found:

1. VPN Issues:
This issue is identical to what @mchitson and others are seeing.

Another user on Discord recently ran into the same VPN issue

2. Connection issues breaking the subscription:
For users that are using the GraphQL API without DataStore, a disruption in the network connection will close the subscription. There are several discussion on Discord on this. One user posted a solution that could work to re-establish the subscription

3. Dev server disconnecting:

With regards to this other comment from @mchitson:

for more info, i also see a comment from webpack after the appsync error:

Not sure if it's react closing the connection or appsync causing react to print out that command.

I haven't been able to reproduce a scenario where a subscription error causes the dev server to disconnect. Given that you're seeing The development server has disconnected in the console, can you confirm if your dev server stopped running at the time you saw this error? It's possible there is a timeout happening here, unrelated to Amplify.

4. Misconfigured subscriptions:

Regarding the following comment from @mchitson

Also I am trying Datastore as I would like my subscriptions to be up to date when a device is temporarily disconnected and reconnected. I am however getting those errors:
DataStore - subscriptionError Connection failed: {"errors":[{"message":"Validation error of type FieldUndefined:

I've run into this before, as well. Without knowing what field is undefined in your app, it's difficult for me to tell you exactly what's happening here. However, when I added auth to an API, and was attempting to subscribe to updates without updating the instantiation of the subscription, I was running into a similar issue because I was not implementing the subscription correctly (i.e. a field was missing).

Hope this helps, @mchitson! Please let us know if your issue is not caused by the above.

[david-mcafee]

Closing this issue. Remediation steps are listed in my previous comment. Please reach out if there are further issues!

[efimk-lu]

Hey @david-mcafee , regarding your comment

2. Connection issues breaking the subscription:
For users that are using the GraphQL API without DataStore, a disruption in the network connection will close the subscription. There are several discussions on Discord on this. One user [posted a solution that could work to re-establish the subscription](https://discord.com/channels/705853757799399426/717109993416949811/827076511928549437)

I'm unable to connect to the discord channel, can you share the answer here?

[david-mcafee]

Hey @david-mcafee , regarding your comment

2. Connection issues breaking the subscription:

For users that are using the GraphQL API without DataStore, a disruption in the network connection will close the subscription. There are several discussions on Discord on this. One user [posted a solution that could work to re-establish the subscription](https://discord.com/channels/705853757799399426/717109993416949811/827076511928549437)

I'm unable to connect to the discord channel, can you share the answer here?

@efimk-lu - it looks like the related Discord chat is no longer available, but the GH issue I linked to also details the same issue. Are you using a VPN? If you are, it's possible that is the cause.

[efimk-lu]

@david-mcafee No VPN, just regular internet connection. The computer sometimes goes into sleep mode, the network gets disconnected and we are unable to subscribe again

[CSHaze]

I am running into this problem as well. It happens on web (computer goes to sleep, network change, etc) and iOS and android (app goes into background).

The only solution seems to be refreshing the page or closing and reopening the app. Neither of these solutions are practical, is there another way to refresh the connection?

[majirosstefan]

I think this should not be closed, it should not forward users to sites outside of Github / and all code samples / workarounds should be posted HERE.

[lehoai]

Got same issue!

[lehoai]

If I leave the tab open for ~20mins, (the internet is still connected) then the subscription is closed.

[AlphaJuliettOmega]

Same issue, no VPN.

Issue should not be closed as there was no mitigation.

The error should also be caught or documentation should be clear that the error will be thrown at unspecified moments in runtime.

[ihtisham-sohail]

Our application is also facing this issue , for us it's mostly triggered when too many subscription requests are coming ( We have a bulk creation feature - upto 1000 DB entries are created and it listens on their creation ).
Not sure if graphQL has a limit on how many concurrent requests it listens

[dannyhw]

We have this issue, no VPN or anything like that. The listed "fixes" are not useful, seems pretty clear that this issue should be re-opened (in my opinion).

This happens intermittently for our subscriptions where the connection is closed by app-sync for seemingly no reason.

Any workarounds for this?

[999roberto999]

I just had this happen while doing some stress testing on our app, so sending a lot of updates. Basically we had 2 out of 6 devices get this error at the exact same moment. Please reopen this issue, it doesn't appear to be related to any of the issues that were outlined above. Not using VPN, Network connection is stable, and the Dev server did not go down. Also I noticed the link to a discord solution for re-subscribing, simply open discord so that doesn't really help at all.

It is alarming to me that this issue is closed when the issue is obviously still happening to a lot of people. I am going to attempt to setup my own system to resubscribe but, any additional guidance would be appreciated as this issue is a pretty big deal for our app and could force us to change cloud services/tools if we don't have any good solutions.

[JandenMa]

This is happening in my team's application, as well.

We use appsync graphql and vue apollo composable, and I see this error quite a bit. From a bit of trial/error, the issue seems to arise from subscribing/unsubscribing in quick succession.

Vue apollo composable handles subscriptions internally in accordance with component lifecycles (from my understanding), so in this case, mounting and unmounting a component that subscribes/unsubscribe quickly will produce this error. If I wait some time in between mounting/unmounting, the error is less common. Not sure if that's actually the root cause or a way to get consistent behavior, just wanted to share in case it could help.

I agree with @Mootook, in my case, mounting and unmounting a component that subscribes/unsubscribes quickly will produce this error.