distr txns: for multi-key updates, reads occasionally seeing inconsistent writetime

[kmuthukk]

For multi-key updates (i.e. keys that may fall on different shards) with distributed transactions enabled, the reads for the two keys in the same select statement are sometimes seeing an inconsistent writetime.

I am seeing this behavior when using the sample app CassandraTransactionalKeyValue.java.

The inserts in this app perform two key updates for the same key prefix. For example, for a key prefix like 100, it updates two keys: 'key:100_1' and 'key:100_2'. These two keys, in the common case, may fall on different shards because the primary key is hash partitioned in the test.

The reads portion of the test then reads both the keys and verifies that their "writetime" (the hybrid time of the write operation) is the same via a SELECT statement of the form:

For example, something like:

  SELECT k, v, writetime(v) 
      FROM <table> 
      WHERE k in ('key_100_1', 'key_100_2');

Issue:

When concurrently running read and write operations, we are occasionally running into a 'writetime' mismatch error for some keys suggesting that we are sometimes reading an inconsistent cut of the two keys.

Could it be that for the two keys we are reading, we are not using the same "read point"?

Sample errors I see during the run:

2018-05-15 14:58:20,756 [FATAL|com.yugabyte.sample.apps.CassandraKeyValue|CassandraTransactionalKeyValue]
Writetime mismatch for key: Key: 6865, value: val:6865, 1526396271311166 vs 1526396300753247

Rough steps to repro.

Note: A RF=1 (replication factor) cluster is sufficient to repro this. First perform about 10K writes (each comprising of a write to two keys):

% ~/yugabyte-1.0.1.0/bin/yb-ctl destroy
% ~/yugabyte-1.0.1.0/bin/yb-ctl --rf 1 create
% java -jar ~/yugabyte-1.0.1.0/java/yb-sample-apps.jar \
       --workload CassandraTransactionalKeyValue \
       --nodes 127.0.0.1:9042  --num_threads_write 8 --num_threads_read 0 \
       --nouuid --num_unique_keys 10000 --num_writes 10000

Now that the keys have been written, in one window we keep running the write operation (which keeps overwriting the same 10K keys in a loop):

# Window 1
% for i in {1..2}; do \
      java -jar ~/yugabyte-1.0.1.0/java/yb-sample-apps.jar \
         --workload CassandraTransactionalKeyValue --nodes 127.0.0.1:9042  \
         --num_threads_write 8 --num_threads_read 0 --nouuid \
         --num_unique_keys 10000 --num_writes 10000; \
    done

And in a second window, run another instance of the same app but in pure read mode. We need to specify the max key (max_written_key) up to which it is OK to read.

# Window 2: Read indefinitely and at random keys with prefix up to 9999.

% java -jar ~/yugabyte-1.0.1.0/java/yb-sample-apps.jar \
      --workload CassandraTransactionalKeyValue --nodes 127.0.0.1:9042 \
      --read_only --num_threads_read 8 --nouuid \
      --max_written_key 9999

[kmuthukk]

[capturing offline discussion here]

@robertpang mentioned that when we read the keys from potentially different tablets, we just need to pick a common read-point and send that along with the read request rather than letting each tablet a make local/independent decision on the read-point.

[robertpang]

Fixed in commit f20bb98.