Query don't copy #380
Query don't copy #380
Conversation
Queries the manifest to avoid copying the whole manifest when taking a snapshot of a penciller to run a query. Change the logging of fold setup in the Bookie to record the actual snapshot time (rather than the uninteresting and fast returning the the function which will request the snapshot). A little tidy to avoid duplicating the ?MAX_LEVELS macro.
|
Testing in standard NHS Riak test cluster on a 2i heavy workload. The result was that the B0017 log showed a total of 1.173ms to snapshot a store prior to the change, and 233ms after the change. |
|
With the database unstressed (i.e. just ten bench workers), the throughput increase was 26.6% with a 10% drop in CPU used. |
| leveled_pmanifest:query_manifest( | ||
| State#state.manifest, StartKey, EndKey); | ||
| QueryManifest -> | ||
| QueryManifest |
There was a problem hiding this comment.
I have a bit of problem to see whether this is compatible with the old solution.
If you register a snapshot, then you now put the query_manifest in the state (line 881). A consecutive fetch_keys will use that query.
But the StartKey and EndKey of the registers snapshot and the fetch might be different? In which case the result of the fetch is different??
More precise, if the fetch is on Query {StartKey = 10, EndKey = 20} and the fetch is on StartKey = 1, EndKey = 8, then the original would use the 1..8 range for fetch, whereas the new one would have saved the query 10...20 in the state and use that. What am I missing?
There was a problem hiding this comment.
If at the point of requesting the snapshot you know what the Query is, then the StartKey/EndKey are provided here. So it is assumed that there will only be one subsequent call to find_keys on that snapshot, and it will be for the same StartKey/EndKey.
There are circumstances where a snapshot is reused for multiple queries (e.g. when doing list buckets), but in this case there is no Query provided when registering the snapshot.
I think it might be worth making the query_manifest a tuple of {SubManifest, StartKey, EndKey} and a guard to crash a subsequent find_keys request which tries a StartKey or EndKey outside of the range. This will then make this requirement explicit.
* Query don't copy Queries the manifest to avoid copying the whole manifest when taking a snapshot of a penciller to run a query. Change the logging of fold setup in the Bookie to record the actual snapshot time (rather than the uninteresting and fast returning the the function which will request the snapshot). A little tidy to avoid duplicating the ?MAX_LEVELS macro. * Clarify log is of snapshot time not fold time * Updates after review
Queries the manifest to avoid copying the whole manifest when taking a snapshot of a penciller to run a query.
Change the logging of fold setup in the Bookie to record the actual snapshot time (rather than the uninteresting and fast returning the the function which will request the snapshot).
A little tidy to avoid duplicating the ?MAX_LEVELS macro.
#379