Snapshot time

[martinsumner]

In this context, a snapshot of the leveled store is a process which can run a read only query returning results which are fixed at the point in time the snapshot was taken.

When taking a snapshot of the penciller to run a query, a copy of the penciller manifest is taken (as with all snapshots). Then as part of the query, in the snapshot penciller process the manifest is slimmed down, to just the part relevant for the query.

There is a Riak user that wishes to run > 10K 2i queries per second. One of the limiting factors for running high volumes of queries is the snapshot time, as the vnode is blocked as the snapshot is taken and every vnode must be involved in 1/N queries.

Currently a snapshot takes approximately 1.2ms. This limits throughput to (1000/1.2)*N -> 2,500 queries per second with a N-val of 3.

The majority of the time is caused by the copy of manifest (not directly when the copy_manifest function is called, but at the point the copied manifest is returned to the calling process). The query to provide the slimmed down copy of the manifest is an order of magnitude faster than this copy.

By running the query against the manifest, and then just copying the slimmed down result to the new snapshot penciller, the time taken to perform a snapshot drops from 1200 microseconds, to 200 microseconds. This changes the throughput limit for 2i queries to This limits throughput to (1000/0.2)*N -> 15,000 queries per second with a N-val of 3.