Sync: Deleted events are not synced well

[AlexeyBarabash]

Description

There is a scenario with high reproduce rate to see that delete event is not synced well.

Steps to Reproduce

1. Create sync chain on brave-core device A
2. Connect to the sync chain device B with code words
3. On the device A open the following tabs:
   brave://bookmarks/
   https://www.bing.com/search?q=BCA-01&qs=n&form=QBLH&sp=-1&pq=bca-01&sc=8-6&sk=&cvid=4CE2DC7674794F3C89E4F55DE31BBD2A
   https://www.bing.com/search?q=BCA-02&qs=n&form=QBRE&sp=-1&pq=bca-02&sc=8-6&sk=&cvid=DA80C45365A54CF4880E4F853CE72332
   https://www.bing.com/search?q=BCA-03&qs=n&form=QBRE&sp=-1&pq=bca-03&sc=8-6&sk=&cvid=5E00C210D471463EB9ECF64D9EC703DA
4. On device A bookmark tabs of BCA-01 and BCA-02 as quick as possible
5. On device B wait until BCA-01 and BCA-02 appear. When this happened as quick as possible, on device A unbookmark BCA-02 tab and bookmark BCA-03 tab

Actual result:

6. On device B BCA-02 is not removed

Sometimes BCA-02 is re-appeared on both devices A and B. Sometimes all works as expected.

Expected result:

6. On device B BCA-02 is removed

Reproduces how often:

not easy, but high reproduce rate

Brave version (brave://version info)

Brave	0.63.0 Chromium: 73.0.3683.39 (Developer Build) (64-bit)
Revision	cc53b0e12fcaf42e4bab8d6c23bd4fb7aae99f6c-refs/branch-heads/3683@{#413}
OS	Linux

Reproducible on current release:

• Does it reproduce on brave-browser dev/beta builds?
  probably yes

[AlexeyBarabash]

Some investigating gives info that js lib may at get-existing-objects

1. give response with some records wrong syncTimestamp order (unsorted)
2. give some records beyond start_at parameter
3. as result of pt 1 and pt 2 lastRecordTimeStamp can be not the latest

The piece of log illustrating situation in issue

[23146:23146:0227/174202.474494:INFO:CONSOLE(21)]
 ""fetch-sync-records" category_names=["BOOKMARKS"] start_at=1551 282 051 883 max_records=1000", ...
...
[23146:23146:0227/174204.665871:INFO:CONSOLE(241)] ""sync-debug" message="got 4 decrypted records in BOOKMARKS after 1551282051883"", source: chrome-extension://nomlkjnggnifocmealianaaiobmebgil/extension/background.js (241)

[23146:23146:0227/174204.670666:INFO:CONSOLE(261)] ""get-existing-objects" category_name="BOOKMARKS" records=[
{
"action":0,"deviceId":{"0":0},"objectId":{"0":112,"1":131,"2":157,"3":167,"4":209,"5":162,"6":154,"7":36,"8":20,"9":68,"10":205,"11":78,"12":142,"13":3,"14":118,"15":100},"bookmark":{"hideInToolbar":false,"isFolder":false,"order":"1.0.0.2","site":{"location":"https://www.bing.com/search?q=BCA-03&qs=n&form=QBRE&sp=-1&pq=bca-03&sc=8-6&sk=&cvid=5E00C210D471463EB9ECF64D9EC703DA",
"title":"BCA-03 - Bing","customTitle":"BCA-03 - Bing","lastAccessedTime":0,"creationTime":0,"favicon":"https://www.bing.com/sa/simg/bing_p_rr_teal_min.ico"},"parentFolderObjectId":{},"prevObjectId":{}},"objectData":"bookmark",
"syncTimestamp":1551 282 111 951
},
{
"action":0,"deviceId":{"0":0},"objectId":{"0":27,"1":228,"2":47,"3":174,"4":42,"5":83,"6":145,"7":13,"8":216,"9":169,"10":107,"11":201,"12":91,"13":255,"14":35,"15":123},"bookmark":{"hideInToolbar":false,"isFolder":false,"order":"1.0.0.1","site":{"location":"https://www.bing.com/search?q=BCA-01&qs=n&form=QBLH&sp=-1&pq=bca-01&sc=8-6&sk=&cvid=4CE2DC7674794F3C89E4F55DE31BBD2A",
"title":"BCA-01 - Bing","customTitle":"BCA-01 - Bing","lastAccessedTime":0,"creationTime":0,"favicon":"https://www.bing.com/sa/simg/bing_p_rr_teal_min.ico"},"parentFolderObjectId":{},"prevObjectId":{}},"objectData":"bookmark",
"syncTimestamp":1551 282 051 877
},
{
"action":2,"deviceId":{"0":0},"objectId":{"0":218,"1":107,"2":242,"3":83,"4":130,"5":181,"6":16,"7":31,"8":63,"9":189,"10":26,"11":62,"12":226,"13":124,"14":72,"15":215},"bookmark":{"hideInToolbar":true,"isFolder":false,"order":"1.0.0.2","site":{"location":"https://www.bing.com/search?q=BCA-02&qs=n&form=QBRE&sp=-1&pq=bca-02&sc=8-6&sk=&cvid=DA80C45365A54CF4880E4F853CE72332",
"title":"BCA-02 - Bing","customTitle":"BCA-02 - Bing","lastAccessedTime":0,"creationTime":0,"favicon":""},"parentFolderObjectId":{},"prevObjectId":{}},"objectData":"bookmark",
"syncTimestamp":1551 282 111 959},
{
"action":0,"deviceId":{"0":0},"objectId":{"0":218,"1":107,"2":242,"3":83,"4":130,"5":181,"6":16,"7":31,"8":63,"9":189,"10":26,"11":62,"12":226,"13":124,"14":72,"15":215},"bookmark":{"hideInToolbar":false,"isFolder":false,"order":"1.0.0.2","site":{"location":"https://www.bing.com/search?q=BCA-02&qs=n&form=QBRE&sp=-1&pq=bca-02&sc=8-6&sk=&cvid=DA80C45365A54CF4880E4F853CE72332",
"title":"BCA-02 - Bing","customTitle":"BCA-02 - Bing","lastAccessedTime":0,"creationTime":0,"favicon":"https://www.bing.com/sa/simg/bing_p_rr_teal_min.ico"},"parentFolderObjectId":{},"prevObjectId":{}},"objectData":"bookmark",
"syncTimestamp":1551 282 051 883
}
] lastRecordTimeStamp=1551282051883 isTruncated=false ", source: chrome-extension://nomlkjnggnifocmealianaaiobmebgil/extension/background.js (261)

And there is a similar log for the case when BCA-02 had been created after delete on both devices.

[AlexeyBarabash]

The situation of re-appearing deleted record

[GeetaSarvadnya]

Verification passed on

Brave	0.63.24 Chromium: 73.0.3683.75 (Official Build) dev (64-bit)
Revision	909ee014fcea6828f9a610e6716145bc0b3ebf4a-refs/branch-heads/3683@{#803}
OS	Windows 10 OS Build 17134.523

• Verified the test plan from Updated DEPS for sync js lib brave-core#1840

Verification passed on

Brave	0.62.49 Chromium: 73.0.3683.86 (Official Build) (64-bit)
Revision	f9b0bec6063ea50ce2b71f5b9abbae7beee319a6-refs/branch-heads/3683@{#858}
OS	Linux

Used the test plan from brave/brave-core#1840

Verified passed with

Brave	0.62.49 Chromium: 73.0.3683.86 (Official Build) (64-bit)
Revision	f9b0bec6063ea50ce2b71f5b9abbae7beee319a6-refs/branch-heads/3683@{#858}
OS	Mac OS X

• Verified test plan from Updated DEPS for sync js lib brave-core#1840 multiple times since issue was not 100% reproducible.