Document deletion not sync'd because user loses access to channel during deletion

[tleyden]

Steps to reproduce:

• Run Sync Gateway commit 760444c with config: https://github.com/couchbaselabs/sync-gateway-testcluster/pull/230/files
• Install todolite-ios on device and simulator (* caveat: had accidentally installed a version of todolite-ios on device that was pointed to todolite-cc database)
• Add new todolist on simulator
• Delete the todo list on device
• Deletion did not propagate to the simulator

Sg logs: https://gist.github.com/tleyden/339d9a0b5d5f8306d494

bulk get request capture from http scoop: https://gist.github.com/tleyden/b536a5dace244b4d8c7e

[tleyden]

Action items:

• Smoke test same test against channel cache version
• See if there are multiple pull replicators running, or why changes requests look strange
• Smoke test deletes @adamcfraser

[tleyden]

Test channel cache version steps:

• Delete todolite from device + simulator
• Change constant to #define kSyncGatewayUrl @"http://ec2-54-145-244-2.compute-1.amazonaws.com:4984/todolite-cc/"
• Deploy to simulator and device
• Verify that todo lists / items match up
• Add new list to device, 1484_list_1
• Verified it sync'd to simulator (lag time: negligible)
• Delete on simulator

Expected: Delete sync'd to device and 1484_list_1 should disappear from list
Actual: 1484_list_1 stayed in list

• Add new list on simulator, 1484_list_2
• Verified it sync'd to device (lag time: negligible) Note: 1484_list_1 still did not delete from device

So it's either a bug in todolite-ios or the Sync Gateway bug is present in both channel cache and distributed index mode.

[tleyden]

Test channel cache version steps:

• Delete todolite from device + simulator
• Change constant to #define kSyncGatewayUrl @"http://ec2-54-145-244-2.compute-1.amazonaws.com:4984/todolite-cc/"
• Deploy to simulator and device
• Verify that todo lists / items match up
• Add new list to device, 1484_list_3
• Verified it sync'd to simulator (lag time: negligible)
• Delete 1484_list_3 on device

Actual: 1484_list_3 never deleted from simulator (waited approx 1 min)
Expected: 1484_list_3 should have deleted from simulator

Logs:

https://gist.github.com/tleyden/9d61bf3dba43d9fba50d

[adamcfraser]

Sanity tested deletes locally - worked as expected. Test details here:
https://gist.github.com/adamcfraser/59f83807617266aa4b36

[hideki]

Run ToDo Lite Android against sync gateway 1.2 and DI version of SG 1.2.
ToDo Lite Android works fine with both of them
couchbaselabs/ToDoLite-Android#64 (comment)

NOTE: ToDo Lite Android does not support the deleting the task list.

[pasin]

I can reproduce what @tleyden commented. I noticed from the log that the device did push the deleted change to the sync-gateway but on the simulator, from the log, it seemed like there is no changes coming.

[pasin]

What do I need to do to setup the Sync-Gateway with the same configuration on my local machine?

[pasin]

Note that: deleting a task can be propagated but deleting a list cannot. Is this could be sync-function issue?

[tleyden]

@pasin For a local setup run Sync Gateway commit 760444c with config: https://github.com/couchbaselabs/sync-gateway-testcluster/pull/230/files -- I can help you in more detail later

[tleyden]

@pasin the delete seems to propagate if you delete app and do a fresh pull (though I need to retest), which means it's not a sync function issue, and that client is correctly pushing the delete

[tleyden]

Testing against Sync Gateway 1.1.1 (http://ec2-54-161-209-25.compute-1.amazonaws.com:4984/todolite-cc/)

• On device, add 1484_list_4
• Verify it sync's to emulator
• On device, delete 1484_list_4

Actual: 1484_list_4 never deleted from simulator (waited approx 1 min)
Expected: 1484_list_4 should have deleted from simulator

Logs: https://gist.github.com/tleyden/71ee8520805315ed78ed

If I delete the app from the simulator and re-install, it shows 1484_list_4 as being deleted.

Todolite ios logs after app reinstall: https://gist.github.com/tleyden/fd338a2068d8728c2c86

[tleyden]

It definitely looks like an issue in todolite-ios at this point, so closing this and leaving couchbaselabs/ToDoLite-iOS#54 open.

[pasin]

I tested with Sync-Gateway 1.1.1 with https://github.com/couchbaselabs/ToDoLite-iOS/blob/master/sync-gateway-config.json, and there is NO issue about deleting the list.

[pasin]

I just reopened the issue as the issue might not be at ToDoLite-iOS.

[tleyden]

Possibly related to #264

[tleyden]

It appears the change is not being propagated because of the way the sync function is written with relation to access grants, not due to a bug in the changes processing. For example, if you listen on the port 4985 changes feed, the change will reliably make it to the changes feed.

The reason the behavior is sporadic (in both 1.1.1 and 1.2.0-rc0) is due to a known race condition in the way the sync function is written, because the following things are happening concurrently inside of SG when a list document such as list-1 is deleted:

1. The user loses access to the list-1 channel, since the document granting access to it is gone
2. The notification function sends new changes about the list-1 channel, including the delete

If the notification happens first, the deletion will appear in the user's changes feed, otherwise it will never appear.

Workaround: if the list document is deleted, explicitly grant access to the channel. Downside: this will bloat couchbase server documents with dead channels.

[tleyden]

Closing as duplicate of #2005