Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replication stops after hitting 400 Invalid attachment error #864

Closed
ajithrnayak opened this issue Aug 12, 2015 · 28 comments
Closed

Replication stops after hitting 400 Invalid attachment error #864

ajithrnayak opened this issue Aug 12, 2015 · 28 comments

Comments

@ajithrnayak
Copy link

Integrated Couchbase-lite 1.1.1 enterprise edition from Index of /couchbase-lite-ios/release/1.1.1/ios/1.1.1-5. And, pull replication with CouchDB stops after replication error hits 400 Invalid attachment.

Earlier versions of CBL were logging a warning as
‖ WARNING: Oops, server is trying to send a compressed attachment, I can't do that yet 18:31:25.548‖ WARNING: CBLMultipartDocumentReader[_id="_design/app"]: received unparseable MIME multipart response: Stopped

[_id="_design/app"]
[_id="_design/Admin"]

Looking at the document IDs, these are _design documents with attachments that are not needed for Mobile device; only Web platform uses it. So, is there a way to skip _design documents from replication? Or, may be, skip document replication by default when invalid attachment is found?

@snej
Copy link
Contributor

snej commented Aug 12, 2015

Could you show the metadata ("_"-prefixed properties) of the design doc that causes the error?

You can use a pull replication filter to skip design docs. The filter needs to be installed on the remote server (CouchDB), then you set its name as the filter property of the CBLReplication.

@ajithrnayak
Copy link
Author

Here's the doc - https://friendpaste.com/4gpK3xqGU811nQDfW4Sptf (err.. they have JS files in them).

This one is another doc:
{"_id":"_design/Admin","_rev":"20-20625ef7f3cadfa43b64d13b9bbb8406","_attachments":{"index.html":{"content_type":"text/html","revpos":20,"digest":"md5-adGSx26dDlY/AMLq0rgQbg==","length":8548,"stub":true},"modules.js":{"content_type":"application/json; charset=utf-8","revpos":20,"digest":"md5-TWunpqPKYw1SmAs+1BTCNA==","length":7417,"stub":true}}}

And, I'll use filter to skip design docs; thank you.

@ajithrnayak
Copy link
Author

‖ WARNING: CBL_Puller[http://someServer/database_one] failed to write {E9D77E4D-FB3B-4BF9-AF42-6E7AF57CDF2F #2-d588bf31a35a77ff43c7afe7be460e97}: status=491
SYNCMGR: active=0; status=2; 0/0; 400 Invalid attachment
‖ WARNING: CBL_Puller[http://someServer/database_two] failed to write {53740b12-b4d2-4a29-884d-afd9c60c3a61 #2-6e2b187e68125b88d877cac10ca63ed4}: status=491
SYNCMGR: active=0; status=2; 0/0; 400 Invalid attachment

I used filter to skip design documents. Now, there are no warnings on versions older than 1.1.1, replication works smoothly.

However, CBL 1.1.1 still stops replication with same 400 Invalid attachment error; with a different documents(not design docs) this time. The documents have following meta data-
E9D77E4D-FB3B-4BF9-AF42-6E7AF57CDF2F
53740b12-b4d2-4a29-884d-afd9c60c3a61

@pasin
Copy link
Contributor

pasin commented Aug 13, 2015

@ajithrnayak,
I believe the 400 Invalid attachment issue that you have encountered should be fixed in the next build of 1.1.1 binary. I will post the link once we have the binary ready.

@ajithrnayak
Copy link
Author

@pasin , thank you. Yeah! I made a build myself from commit 5b858e3 and I couldn't see 400 invalid attachment issue, nor any warnings.

@pasin
Copy link
Contributor

pasin commented Aug 13, 2015

Thanks for letting us know. Also the new 1.1.1 build is ready if you are interested in.

http://latestbuilds.hq.couchbase.com/couchbase-lite-ios/release/1.1.1/ios/1.1.1-6

@pasin pasin closed this as completed Aug 13, 2015
@ajithrnayak
Copy link
Author

@pasin I'm still seeing this issue. Scenario goes like:

  • App goes to background
  • Device goes to sleep
  • make 2 changes and '-save' on other device (note: making 1 change works just fine)
  • Wake device and bring app to foreground
  • replication.lastError throws 400 Invalid attachment

After this pull doesn't works for subsequent changes and push works fine.

I'm using : couchbase-lite-ios/release/1.1.1/ios/1.1.1-11 (enterprise)
_attachment looks like this

"64863b_1418896327507.png": {
"content_type": "image/png",
"revpos": 2,
"digest": "md5-g2zwRcmJFPoBlJIYJykSDg==",
"length": 111221,
"stub": true
}

@pasin
Copy link
Contributor

pasin commented Aug 27, 2015

@ajithrnayak could you please post the complete JSON document?

I'm opening the issue so that this will not get missed in the next sprint.

@pasin pasin reopened this Aug 27, 2015
@ajithrnayak
Copy link
Author

@pasin
Copy link
Contributor

pasin commented Aug 27, 2015

The document itself looks fine to me. Was the document deleted before the app went to background?

@pasin
Copy link
Contributor

pasin commented Aug 27, 2015

Also are there any warning messages shown on the log? They might give us some clues.

@ajithrnayak
Copy link
Author

This is the log on launching app. I checked documents and they look fine (same as above).

SYNCMGR: active=0; status=1; 0/0; (null)
SYNCMGR: active=0; status=1; 0/0; (null)
SYNCMGR: active=0; status=1; 0/0; (null)
SYNCMGR: active=1; status=3; 0/0; (null)
WARNING: CBL_Puller[http://server/database] failed to write {E00E42B72CEB48A3862380B45F5A79C7 #5-72798a76630ae392395eec265fed2e57}: status=491
SYNCMGR: active=1; status=3; 0/0; (null)
SYNCMGR: active=1; status=3; 0/0; (null)
SYNCMGR: active=0; status=2; 0/0; (null)
SYNCMGR: active=1; status=3; 4/10; (null)
SYNCMGR: active=1; status=3; 4/24; (null)
SYNCMGR: active=1; status=3; 0/1; (null)
SYNCMGR: active=0; status=2; 0/0; 400 Invalid attachment
WARNING: CBL_Puller[http://server/database] failed to write {EB6ADBDFEE3E49989A8682BF2C63A9B5 #12-51de459f827b83b9503eebc09e9867e0}: status=491
SYNCMGR: active=0; status=2; 0/0; 400 Invalid attachment

@ajithrnayak
Copy link
Author

Was the document deleted before the app went to background?

No, it was not deleted.

@pasin
Copy link
Contributor

pasin commented Aug 27, 2015

I couldn't reproduce the issue with the same revision and attachment structure as you have posted (If I understand correctly that is the document from the Sync-Gateway).

To be sure that I really have the same structure, I would need to see the JSON document that has the issue on both sides ([1] the device that tried to pull the document side and [2] the sync-gateway side). Could you help me on this?

To get the document on the device side, you could just do the followings:

CBLDocument doc = [db1 documentWithID:@"document id of the document that has the issue"];
NSLog(@"%@", doc.properties);

@ajithrnayak
Copy link
Author

@pasin No, replication happens with CouchDB.

Here's the device side document - https://friendpaste.com/BmMcxVtbgZnRmlkKI2dXs

@pasin
Copy link
Contributor

pasin commented Aug 28, 2015

Look like digests are different (SHA1 and MD5). Not sue if we have the code to handle this case. Will need to look into the detail.

@ajithrnayak
Copy link
Author

Is different digest a problem? It doesn't fails always; all the other documents (those that sync without a problem) also have different digest.

@pasin
Copy link
Contributor

pasin commented Aug 28, 2015

Just checked the code and there is no digest check for this case (stub = true).

I'm confused with the revision id. On device, it has 4-4ca58886845ee968e5be0dba27ff645c and on couch db, it has 3-d80d38b0124a826e1cd3e7b5456473b8. You explained earlier that you updated the document on another device twice times and tried to sync the document back to the original device. Hence at least on the original device, the generation number (the prefix number at the beginning of the rev id) should be less than the one of the CouchDB. Do I misunderstand anything here? Is it possible we are looking at the wrong document?

@ajithrnayak
Copy link
Author

Ah! I apologies. Just reproduced issue and here's the document:

rev-1 on CouchDB: https://friendpaste.com/BmMcxVtbgZnRmlkKIjU2B
rev-1 on Device : https://friendpaste.com/BmMcxVtbgZnRmlkKISDdz
and after 2 changes,
rev-3 on CouchDB : https://friendpaste.com/7fuz9RELwNfqImn2Zahrog

And, why would I get "400 Invalid attachment" error?

@ajithrnayak
Copy link
Author

@pasin & @snej We are up for a release, but held by this issue. I'm trying to provide more info on this.

Enabled logging for above mentioned scenario and logs(after app is launched again) are here https://friendpaste.com/2BayCv2bZVFNu4XO2qsUUB

You could notice sync hitting error and stopping replication logs from line number 784 to 793. I've quoted the same below.

‖ SyncVerbose: CBL_Puller[http://local_server/nl_wolterendros_denhaag_04140045_sdk_escamp] inserting 5D7471DD62E942B0AD3E48B7A6EF1BDC ["3-b02f72304d8ba69762882bb47707f81a", "2-bcda3a41d759213e72be4df2a627a44e", "1-0c14bc709abdcc1a22d34b61e29213cb"]
‖ WARNING: CBL_Puller[http://local_server/nl_wolterendros_denhaag_04140045_sdk_escamp] failed to write {5D7471DD62E942B0AD3E48B7A6EF1BDC #3-b02f72304d8ba69762882bb47707f81a}: status=491
‖ Sync: CBL_Puller[http://local_server/nl_wolterendros_denhaag_04140045_sdk_escamp] Progress: set error = 400 Invalid attachment
‖ Sync: CBL_Puller[http://local_server/nl_wolterendros_denhaag_04140045_sdk_escamp] STOPPING...
‖ Sync: Stopping 0 remote requests
‖ SyncVerbose: CBL_Puller[http://local_server/nl_wolterendros_denhaag_04140045_sdk_escamp] finished inserting 1 revisions
‖ Sync: CBL_Puller[http://local_server/nl_wolterendros_denhaag_04140045_sdk_escamp] inserted 1 revs in 0.002 sec (572.7/sec)
‖ Sync: CBL_Puller[http://local_server/nl_wolterendros_denhaag_04140045_sdk_escamp] Progress: set active = 0
‖ SyncVerbose: CBL_Puller[http://local_server/nl_wolterendros_denhaag_04140045_sdk_escamp]: postProgressChanged (0/1, active=0 (batch=0, net=0), online=1)
‖ Sync: CBL_Puller[http://local_server/nl_wolterendros_denhaag_04140045_sdk_escamp] STOPPED

@pasin
Copy link
Contributor

pasin commented Sep 1, 2015

I haven't had a chance to test it with CouchDB yet. I will try to reproduce the issue with CouchDB today. Will let you know.

@pasin
Copy link
Contributor

pasin commented Sep 2, 2015

Yes, I could reproduce the issue with CouchDB.

Local:

{
    "_attachments" =     {
        myattachment =         {
            "content_type" = "text/plain; charset=utf-8";
            digest = "sha1-oaDsDkUDon9gAShhMF9l7cxugW8=";
            length = 51200;
            revpos = 1;
            stub = 1;
        };
    };
    "_id" = doc;
    "_rev" = "1-119725eeb7585545cc849fda550d09b8";
    foo = bar;
}

CouchDB:

{
   "_id": "doc",
   "_rev": "3-9088185d9a975e429e1366bf1463e4d7",
   "foo": "bar3",
   "_attachments": {
       "myattachment": {
           "content_type": "text/plain; charset=utf-8",
           "revpos": 1,
           "digest": "md5-W3qvModIgI6HVCj3N0O4IQ==",
           "length": 51200,
           "stub": true
       }
   }
}

Local Pull replication (GET db/doc?rev....) response:

{
  "_id": "doc",
  "_rev": "3-9088185d9a975e429e1366bf1463e4d7",
  "foo": "bar3",
  "_revisions": {
    "start": 3,
    "ids": [
      "9088185d9a975e429e1366bf1463e4d7",
      "3a7f046e6be0574876548e780b032ccf",
      "119725eeb7585545cc849fda550d09b8"
    ]
  },
  "_attachments": {
    "myattachment": {
      "content_type": "text/plain; charset=utf-8",
      "revpos": 1,
      "digest": "md5-W3qvModIgI6HVCj3N0O4IQ==",
      "length": 51200,
      "stub": true,
      "encoding": "gzip",
      "encoded_length": 85
    }
  }
}

@pasin
Copy link
Contributor

pasin commented Sep 2, 2015

This issue has already been fixed in the master branch but the fixes didn't get included in v1.1.1.

Issues #802, #843

Fixes a20e014, aefde4a

@ajithrnayak
Copy link
Author

Alright, glad to hear that. Thank you, @pasin. I can make a build myself from master. But, is this going to be included in v.1.1.1 anytime soon?

@ajithrnayak
Copy link
Author

I see a20e014 (i,e #802) already on v.1.1.1 list of fixes.

@pasin
Copy link
Contributor

pasin commented Sep 2, 2015

The fix list is compiled from the issues that are tagged with 1.1.1. It was supposed to go in to v1.1.1 based on the tag but we missed that commits when cherry-picking the fixes. I'm checking now if it's still possible to go into 1.1.1 or it's too late.

@ajithrnayak
Copy link
Author

I couldn't reproduce this on a build from master branch. Cheers!

@pasin
Copy link
Contributor

pasin commented Sep 2, 2015

We have just included those fixes in 1.1.1 release. You can download the latest build from here. You may want to try it out.

In the current master branch, we have changed the database file directory layout. It will automatically upgrade from the old structure to the new structure. However 1.1.1 version will not be able to recognize that new structure. This means that switching back from master to 1.1.1 binary may require you to delete your development app first so the 1.1.1 database can be regenerated.

Thanks a lot for reporting this, otherwise we may miss this fix in 1.1.1 release.

@pasin pasin closed this as completed Sep 2, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants