App not updating users location with app closed #2508
Comments
|
Does rnbgl still upload location data points if the app is completely killed? I vaguely remember talking about this several weeks ago ... I think the answer then was ... No, but I'm not sure if I remember correctly. |
|
If user @ally leaves bot location "Home Sweet Home" and opens the app (the same time that @miranda shows that we left "Home Sweet Home") the widget will show that she is still at "Home Sweet Home". She sends the app to the background then foreground and you could tell that the widget refreshed but was still showing her at "Home Sweet Home". I then had her send it to the background and leave it for a few seconds. She brought it back to foreground and it updated to show she had left "Home Sweet Home". I now have 3 users that are experiencing this issue. @nate, @don & @ally |
|
rnbgl doesn't send updates if the app is killed (at least not until the user re-opens the app), but if it's merely backgrounded (as this ticket suggests), then it should still send updates as expected. |
|
This may have something to do with: #2510 I say this because, if the server thinks the user is online, it won't send any push notifications. Anyway... something to possibly consider. |
|
I think you are conflating things.
The server doesn't do this. If the business logic says to 'send a push notification', the server doesn't check whether the account is online or not. It will just send it. However, if the app for that account is in the foreground when the push notification is sent, iOS won't show the push notification. They may be other cases when iOS may also choose not to show the push notification. I'm not saying that this is the reason, but it cannot be ruled out. |
|
googles the word "conflating" 😆 |
|
When I read the description, nothing stands out as a probable cause. I'm not sure where to start looking. If this was a Staging ticket, I'd ask for confirmation whether the device actually uploaded a location data point (in the debugging area). That's a good way to bisect the problem. Does this problem happen everytime both you and Ally leave/enter a bot? Sometimes? Rarely? Also, are you able to reproduce this on Staging? |
|
@bengtan This issue is everytime, new users (@ally, @nate, @sydnie) are not generating push notifications for their enter/exit of any bots since downloading the app. The only time they generate a push is if they open the app. I have not been able to reproduce this issue on Staging with user @don. It appears to be working fine on Staging. Update 6/29: Per user @Tara: @nate & @sydnie are now generating push notifications. |
|
I deleted & re-installed the app on @ally’s device. We closed the app and I drove her to the pool. We both received a push for her exiting “Home Sweet Home” but when we got to the pool there is no push for at the "Mooreland pool" after waiting at least 5 minutes. When we returned to "Home Sweet Home" we received push that she arrived. So this is working better after the re-install. |
|
I did some troubleshooting with @mstidham this afternoon. I was able to pin down a reason for the documented behavior in this ticket in one instance, but I can't say for sure what it means or that this is the definitive cause. According to @mstidham, user @ally (b38697f0-75a6-11e8-a961-0a580a020460) was with her at the bot location "Home Sweet Home" (53e63120-c2e7-11e6-be1e-0e06eef9e066). They both left the bot area around 15:54 CT (20:54Z) on 2018-07-02. No push notification was sent to indicate that @ally had left the bot. I double-checked that the bot had geofence enabled and that @ally was subscribed as a guest. Then I looked at the bot events table and saw that the last event for @ally was on 2018-07-01. Looking deeper into the location table, I saw that there was a big gap in the location data for @ally. There was a datapoint at 17:31:10Z and then nothing until 20:59:37Z. I did the distance calculations by hand and discovered that the first datapoint was about 20Km from the bot and the second was about 2.1Km. Both are well outside the 100m radius of the bot. Clearly, what happened, in this case, is that the server never sent a push notification for @ally leaving the bot, because as far as the server is concerned, they never entered it. @ally came and went from the bot in that 3.5 hour span when her phone was not uploading locations. |
thescurry
changed the title
|
@bengtan - Possibly something is happening on the client side that's preventing location upload? |
|
Did one test run. Staging 3.1.3.
Inspected the database. Location data points were uploaded throughout the entire journey, including whilst I was entering/leaving B. Attempt to reproduce was unsuccessful. I'll have to try again another time, maybe with an older version. Off-topic: Even though I killed the app, it still uploaded location afterwards. |
|
A status update: @mstidham informed me of two situations when this ticket (or at least, symptoms similar to the OP) apparently occurred again.
Upon investigating 1 (for Prod), I confirmed that, yes, location data points were being dropped (or delayed significantly) on Prod on version 2.6.8. However, I also observed that #2510 was also active at the time. I have a suspicion that #2508 and #2510 are related. It's not that #2510 causes #2508 (or vice versa), but that they are both caused (in a convoluted way) by the same thing. I am cautiously optimistic that the fix for #2510 (deployed as of version 3.1.2) also fixes #2508. Upon investigating 1 (for Staging on version 3.1.3), I observed that, AFAICT, the location data points look fine. However, the debouncing was incorrectly not triggered. I've posted hippware/wocky#1713 as a result. So situation 1 indicates that it does happen on 2.6.8 (Prod) but it wasn't observed on 3.1.3 (Staging). Upon investigating 2 (Staging version 3.1.3), it turned out to be hippware/wocky#1713 again. In summary, this bug is confirmed to exist for version 2.6.8. I'm cautiously optimistic that it is fixed in 3.1.2/3.1.3 by the same fix that fixed #2510. @mstidham alerted me to two instances when symptoms similar to the OP occurred, but upon investigation, it was either an old version (Prod 2.6.8) or due to another issue (hippware/wocky#1713). Hence, I'm moving this ticket to 'Ready for QA' and @mstidham and I will monitor it. |
|
This issue has crept back up on Version: 3.6.1. |
|
@mstidham: Staging or Prod? Can you give me a description of what happened, what you expected, what you observed? With timestamps and/or (expected) locations and/or screenshots please. I'll have a look tomorrow. |
|
Time of screenshot: 10/1/18 11:16 PM UTC
Staging 3.6.1:
|
|
@bengtan Maybe this regression happens after my fix to redirect user to current location after app re-launch (i.e. by setting location to null when user goes to background) |
|
As of 10/2/18 approximately 5:00 PM UTC I started receiving push notifications again for Prod. @thescurry also seems to be working correctly again on Staging this afternoon. (10/2/18 approx. 11:30 PM UTC) |
|
Yeah... it's really weird. Yesterday frustratingly things didn't work well... today they work great. Pavel did your codepush today include any fixes for this? 🤔 |
|
I think so
… On 3 Oct 2018, at 02:31, Steve Curry ***@***.***> wrote:
Yeah... it's really weird. Yesterday frustratingly things didn't work well... today they work great. Pavel did your codepush today include any fixes for this? 🤔
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
There was an issue with the server side (Staging and Prod) which resulted in 100% CPU usage and slow/laggy performance, and possibly dropped stuff. It started around 2018-10-02 15:00 UTC and lasted for about 4.5 hours, approximately. It might be related, or not. Anyway, I'll go dig around in the DBs. |
|
A.
Miranda's location data points look fine for Staging (as also suggested by the screenshot). The push notifications for Staging according to the DB: ... which match what's in the screenshot. Location data points for Prod ... there's a 3.5+ hour gap when no location data points were uploaded. ... So it looks like the app isn't sending location data points to Prod. I'd call it confirmed. However, if I look at the traffic logs, I can see that she closed/backgrounded her app at 19:54:50 and reopened/foregrounded it at 23:46:53. If I include this with 3.5 hour gap, I get: ... which is suspiciously coincidental. When the app is closed, there are no more location data points. When the app is opened, a location data point is uploaded. @mstidham: Do you have Location access set to 'While Using The App' (for Prod)? B. I looked into both cases where @thescurry was at a location but a (push) notification didn't fire until about 4 minutes later (possibly coinciding with him opening the app). This was for the bots 'Blue Bottle - Robertson Blvd' ( This wasn't a case of location data points not being uploaded. They were being uploaded. They were both cases of the 30-second debounce algorithm and it's shortcomings. In both cases, the 30-second timer had started and the server was waiting for one more location data point to confirm it, but the phone didn't send any more points because it was stationary. When @thescurry opened his phone, this caused a location data point to be uploaded, hence satisfying the timer. (I have observed that, generally, when the app is open/foregrounded, it usually sends one location data point, even if it is stationary.) So ... it's a known problem ... that with the 30 second debounce rule, if the phone isn't sending any more data points, the rule will fail. We've mitigated that recently by putting in some tweaks as shortcuts, but occasionally the shortcuts won't work and the 30 second rule is used as the fallback ... together with it's shortcomings. Note for future reference: 2018-10-01 21:05:57 UTC scurry is at Four Seasons |
|
Diagnosed. Not related to anything @aksonov changed, nor the recent server degradation/outage. |
|
I did not have my Prod setting set to While Using, it was set to Always. |
Okay. Then I'll have to find another explanation. |
|
I have no issues with another release going to the app store, especially if things are improved. That's definitely an option. |
|
I can say that I was unstuck on Prod after the codepush so that is an improvement, however there are some locations that I went to multiple times that never generated push notifications and I was clearly within the radius. So I do still have some concern. |
|
@mstidham when you have locations where you arrive and don't get a push notifcation, can you note the location and date/time (this might be a lot of note taking before we squash all these bugs)? Hard to say if these are releated, could be that we're also hitting the issue where you arrive and don't generate enough location updates to please the debounce. @bengtan thoughts on this? I can do the same, we can just post these notes into QA and hopefully help Beng track some of these corner cases down. |
That was the plan and it totally slipped my mind...my bad. Firing it off now... |
|
@thescurry @bengtan in terms of a new AppStore release we've included a lot of potentially breaking changes (mobx, MST, typescript, etc) to 3.9.4 that need to be very thoroughly tested before being considered release-worthy. |
|
@southerneer correct, not in a rush to push anything to the appstore atm. 👍 |
Yeah, please. This is not the only RNBGL issue we have. There is also #2984. So whenever you encounter something unexpected, it could be this ticket, or it could be #2984, or it could be something new. QA won't be able to disambiguate the two. I usually have to look in the DB and trace the location data points to disambiguate. (cc @thescurry @mstidham) |
|
Quoting @thescurry:
It's been merged into 3.9.4. However, 3.9.4 has a lot of changes so it won't make it to the App Store for a while. Nor should 3.9.4 be released via codepush anytime soon. So ... if the situation warrants, we could codepush just this fix as a 3.9.3.1 ... but I think we should at least do some more field testing on it. Maybe a few days? Optionally, we could wait for #2984 and do them both (and other stuff?) as a 3.9.3.1 release (and a codepush). |
|
This wasn't moved back to 'Up Next' previously. However, a fix for this was deployed in 3.9.4 so it is again 'Ready for QA'. Also, fyi, @mstidham has been testing a codepushed version of this fix. |
|
@mstidham encountered what seems to be another instance of this. I've looked at it and can't explain it yet. I want to have another look later so I'm writing some stuff for historical reference. User: at-miranda
Re: the screenshot At 2018-11-06 20:15:58.719623, the user left the bot. We went out to have lunch. Lunch took about an hour? So we had returned, and entered the bot, about an hour afterwards. At 2018-11-06 21:55:45.905592, the user opened the app and caused a location data point to be uploaded. This triggered an entry event. Expected result: Location data points should have been uploaded which would have resulted in an entry event about an hour after 20:15. Observerd result: The next location data point was uploaded at 21:55, about 30-40 minutes later than expected. It might have been even later if the user had not opened the app. |
|
Also noting that I'm on Codepush Staging-Pavel |
Specifically, https://appcenter.ms/orgs/hippware/apps/tinyrobot-2/distribute/code-push/StagingPavel/v28 |
|
FYI. Don encountered a new instance on Prod. In this case, there was a log. I examined it and didn't come up with an explanation. Device mysteriously stops uploading location |
|
Got a reply from RNBGL maintainer and spun out an action ticket: Call RNBGL ready() on every app start (even from background) #3071 |
|
Steps to reproduce this latest cause: 'PowerHome kill' method
Successfully tested on iOS simulator (iPhone 7, iOS 11.4) and iPhone 6 (iOS 11.2.1) Apparently this technique (which I'm calling 'powerhome kill') kills all apps on the phone. I got this technique off the Internet, but I also read somewhere that it doesn't work anymore on later/latest iOS so it might only work on iOS 11.x. This is unconfirmed. I also tried ... At step 3, press the 'X' / Cancel software button instead of the Home button. That didn't successfully reproduce the problem. A conventional 'double-tap-home' kill didn't reproduce the problem either. DiscussionI think what's happening is that ... under some conditions, iOS kills the app (ie. out of memory or whatever) and RNBGL/the-app doesn't recover. It must be quite rare. I'll try to implement a fix as ticket #3071. |
|
Thanks for the update Beng. |
|
Found another test case. The second one below. Scenario 1:
Unsuccessful in reproducing. Scenario 2:
Successful in reproducing. So rebooting the phone can also reproduce this problem. It's something additional that I will have to test for. |
|
Back to 'Waiting for Deployment' via #3071 |
|
Version 3.9.3.1 is available on StagingBeng with a fix for this. See #3152 (comment) QA NotesIf you want to artificially reproduce this issue, you can do one of the following:
[1] PowerHome kill: Once you start moving, the app takes about 5-10 minutes to start up and begin uploading location again. Anecdotally, it takes about 5 minutes to recover if the app is swiped away beforehand (1 and 3), and about 10 minutes to recover if the app is not swiped away beforehand (2 and 4). However, I haven't timed this so don't quote me on it. It's probably safer just to give it 10 minutes. The corollary to the 5-10 minute recovery window is that, if the app does get killed by the OS for normal reasons, there will be a 5-10 minute gap where nothing is uploaded. This should be considered to be normal behaviour. The litmus test is whether the app recovers after 5-10 minutes. In fact, in the following scenario:
The app might not even recover at all (since the user is stationary) until the user starts moving again. There is not much more that we can do about this. (Unless we want to start delving deep in React Native startup code.) Also, another subtle thing you can look out for. If you artifically reproduce this issue via 1-4 above, and if the app has recovered, then when you open the app, it shows the Home screen (ie. with the map) straight away instead of showing the loading screen for a minute or two. So you can discern whether the app has started in the background by observing whether the loading screen appears or not. Moving to 'Ready for QA' (presuming QA tries out Staging 3.9.3.1) |
|
Prod 3.9.2 + codepush Beng |
|
@mstidham: This has been on Prod (as 3.9.3.1) for a while now. Is it good enough to be considered 'Verified on Prod'-ed? |
|
It's working perfectly for me. Verified on Prod |
App not updating users location with app closed
Observed Result:
User @ally left "Home Sweet Home" bot at the same time as @miranda. There was no push notification generated for @ally. @ally did receive a push notification that @miranda left "Home Sweet Home" so she is able to receive them. When we arrived at the "Mooreland pool" again no push notification was received for @ally's arrival. We then opened the app and @miranda received push for @ally leaving "Home Sweet Home" and then arriving "Mooreland pool". Confirmed that user had background refresh on and cellular data on.
Steps to reproduce:
Related Screenshots/Video Links:
The text was updated successfully, but these errors were encountered: