API: Public method for inserting existing revisions (putExistingRevisionWithProperties:...)

[hermanccw]

Hi there,

Scenario:
In production, some of our user has quite a bit of data built up in our app (like more than 8000 documents) and from time to time, they need to reinstall our app (either because of a new phone purchase or remove app by accident).

As you can imagine, it takes awhile to sync more than 8000 document down to a mobile device (awhile meaning more than 1 minute, which is eternity nowadays). This became a problem for us since our app require certain kind of document to be functional (think type = "user", type = "group"). So if a user got frustrated and exit our app before these document comes down through replication, they are stuck in a state that they can't do anything when they reopen our app again. Yes, we can tell them to have the app open for awhile until these document shows up (which is what we've been doing), but that's hardly a good user experience.

So for us, the problem is really trying to seed some important data from our server when the user first enter into the app and sign in/up. Here is what we came up with.

Solution:
We created an endpoint in our api that will return something like this

"db_name_1" =     (
{
            "_id" = "importDoc1ID";
            "_rev" = "importDoc1CurrentRev";
            "_revisions" =             (
                "2-9457a27fe1448fa5a94f2f7c7eb09456",
                "1-bd5bab3f1b4ee779345fc01d5951b261"
            );
            other props.....
},
{
            "_id" = "importDoc2ID";
            "_rev" = "importDoc2CurrentRev";
            "_revisions" =             (
                "2-9457a27fe1448fa5a94f2f7c7eb09456",
                "1-bd5bab3f1b4ee779345fc01d5951b261"
            );
            other props.....
}
),
"db_name_2" =     (
),
........

The keys are the database names that we have in our app, and within each key, we includes all the import document that we need for our app to be functional right out of the gate (note the _revision prop is in reverse order). Then, we take this JSON and feed this into the function below.

- (void) insertImportData {
    NSDictionary *seedData = JSON_FROM_SERVER;
    NSError *__autoreleasing error;
    for (NSString *dbName in seedData) {
        // create group with group id
        if ([[CBLManager sharedInstance] databaseExistsNamed:dbName]) {
            // if db exist already, keep it
            continue;
        }

        CBLDatabase *db = [[CBLManager sharedInstance] databaseNamed:dbName error:&error];
        if (!error && db) {
            id dbData = seedData[dbName];
            if (dbData && [dbData isKindOfClass:[NSArray class]]) {
                for (NSDictionary *data in dbData) {
                    NSMutableDictionary *dataProps = [NSMutableDictionary dictionaryWithDictionary:data];
                    [dataProps removeObjectForKey:@"_id"];
                    [dataProps removeObjectForKey:@"_rev"];
                    [dataProps removeObjectForKey:@"_revisions"];

                    id revision = [NSClassFromString(@"CBL_MutableRevision") alloc];
                    revision = objc_msgSend(revision, NSSelectorFromString(@"initWithDocID:revID:deleted:"), data[@"_id"], data[@"_rev"], NO);
                    if (revision) {
                        objc_msgSend(revision, NSSelectorFromString(@"setProperties:"), [dataProps copy]);
                    }
                    NSArray *history = [data[@"_revisions"] copy];
                    objc_msgSend(db, NSSelectorFromString(@"forceInsert:revisionHistory:source:"), [revision copy], history, nil);
                }
            }
        }
    }
}

Basically, what we did was just call forceInsert:revisionHistory:source with some objc runtime magic on the seedData.

Enhancement request:
So, having said all that. The enhancement request is to have way in CBL to seed external data before any replication happens. It'll be great to have a single call to do all these, but maybe as a stop gap, we can open up CBL_MutableRevision and "forceInsert:revisionHistory:source: so that we don't have to use objc_msgSend in iOS.

[snej]

What we usually recommend in this situation is assigning the urgent docs to a special channel, and first replicating with just that channel. Then when that replication completes, you can start another pull for all the docs.

(If you're not using Sync Gateway, you can probably do the same thing by using a filter function on your CouchDB/Cloudant server.)

But I do agree that -forceInsert should be available in the public API; though probably with a better name.

[hermanccw]

Thanks @snej , forgot about filters (we are using CouchDB). It would work for us if the critical docs we need are pretty static, but for our case, we actually need the latest activity that show up in a database, which would be kind of hard to write a filter function for.

Anyways, good that you think opening up -forceInsert is a good idea, hope to see this going into master soon.

[snej]

I've implemented it in Obj-C; I need to write up the cross-platform API proposal. What I've got is a method on CBLDocument that looks like:

/** Adds an existing revision copied from another database. Unlike a normal insertion, this does
    not assign a new revision ID; instead the revision's ID must be given. The revision's history
    (ancestry) must be given, which can put it anywhere in the revision tree. It's not an error if
    the revision already exists locally; it will just be ignored.

    This is not an operation that clients normally perform; it's used by the replicator.
    You might want to use it if you're pre-loading a database with canned content, or if you're
    implementing some new kind of replicator that transfers revisions from another database.
    @param properties  The properties of the revision (metadata like _id and _rev will be ignored)
    @param revIDs  The revision history in the form of an array of revision-ID strings, in
                    reverse chronological order. The first item must be the new revision's ID.
                    Following items are its parent's ID, etc.
    @param sourceURL  The URL of the database this revision came from, if any. (This value shows
                    up in the CBLDatabaseChange triggered by this insertion, and can help clients
                    decide whether the change is local or not.)
    @param outError  Error information will be stored here if the insertion fails.
    @return  YES on success, NO on failure. */
- (BOOL) putExistingRevisionWithProperties: (NSDictionary*)properties
                           revisionHistory: (NSArray*)revIDs
                                   fromURL: (NSURL*)sourceURL
                                     error: (NSError**)outError;

[suzhaev]

@snej What will happen if i insert only one last revision to revisionHistory parameter? This is normal?