Obsolete rows left in view index when documentType is used (1.1.1)

[aibrahim16]

I've an issue where some documents passes through the map function with its userProperties, but when i try to get the document itself, i don't found the userProperties and find that it has _removed = true but not removed from the CBLView, when i tried to make get all documents from REST i didn't get those removed documents.

I'm using CBL 1.1.1

[snej]

If you lose access to a document on the server (because you lose access to all the channels it's in), the replicator will pull one final stub revision containing nothing but "_removed":true, so that the client knows the document is gone.

It shouldn't be a problem that these show up to your map function, because they don't contain any data that would trigger your map function to emit anything.

[aibrahim16]

But what i get in the map function is a document with all the properties which causes the map function to emit it. While if i tried to get the document itself from the CBLQueryRow i find it without userProperties and with _removed = true.

[aibrahim16]

I've just noticed that this problem happens only when i set the documentType property for the view.

[snej]

That sounds like a bug, then. As though when the documentType doesn't match, the doc is skipped but the obsolete rows don't get removed.

Could you try the latest 1.2 release candidate, and see if it still happens? We're about to release 1.2, and a lot of things have changed.

[aibrahim16]

I've tried 1.2.0-112, but the issue still exists in this release.

[snej]

This is using the default storage engine (SQLite)?
I'll check whether our unit tests cover this situation, and if not, add one.

[aibrahim16]

Yes it uses the default storage engine

[snej]

We've got a unit test that checks this exact situation. So maybe I'm not understanding exactly what's going on in your case.

Could you describe the problem in detail, as a series of steps?

[aibrahim16]

From server-side i'm not aware of much details, but there are updates that happens periodically to the documents, those updates may revoke the access of specific documents to users, but the documents are not deleted and can be accessed from admin port.

From client-side:

• I create the database and set the authenticator for both pull and push replications and set channels
• Create the view like this

let view = database.viewNamed("VIEW_NAME")
        view.documentType = "DOC_TYPE"        
        if view.mapBlock == nil {
            view.setMapBlock({ (doc, emit) in
                let (key, value) = rowItemFromDocument(doc)
                emit(key, value)
            }, version: viewVersion)
        }

• In the map function i receive documents with all its properties.
• But after that when i create a liveQuery, i find that some of those rows refers to documents with _removed = true
• If i removed this line view.documentType = "DOC_TYPE", it works fine

[snej]

Your map block might be throwing an exception if it doesn't find the properties it's expecting … I think that could cause symptoms like what you're seeing (it'll abort the update of the view index, leaving it out of date.) Have you checked the logs for any warnings about exceptions?

[aibrahim16]

I already check for the properties i'm expecting before emitting, the map function is actually like this

{ (doc, emit) in
                if let (key, value) = rowItemFromDocument(document) {
                    emit(key, value)
                }
}

i've also checked the logs and there is no warning or exception

[snej]

This is very mysterious. Could you enable ViewVerbose and QueryVerbose logging (via CBLManager) and then reproduce the problem, and let me look at the logs? (I'd need to know the _id of the document with the problem.)

[aibrahim16]

Those are the ids for some of the current affected documents: Brand::442, Brand::541, Brand::538

[snej]

Can you post the logs too (as requested above)?

[aibrahim16]

Aha sorry, here is the logs

logs.txt

[snej]

Sorry, I can't figure out the problem from those logs. I see the document getting indexed once, then returned in queries several times. Could you please also turn on View and CBLDatabase logging (in addition to ViewVerbose and QueryVerbose)? Then run it again and capture the logs. Thanks!

[aibrahim16]

Currently i'm not able to reproduce the issue, i'll try again and post the logs once i'm able to reproduce it.

[zgramana]

@aibrahim16 please re-file if this happens again.