[Jepsen][YCQL] Consistency check failure for improved bank workload

[qvad]

Jira Link: DB-923
I've extended coverage for YCQL bank workload, and now it also trigger inserts with same invariant.
Pull request with improved bank workload yugabyte/jepsen#55

Here is source code for inserts. On each dice == "insert" there is automatically incremented counter, so there is no concurrent runs here and each insert statement gets unique value, even when operation is failed.
May occur seven w/o any nemesis running

(let [{:keys [from to amount]} (:value op)
        dice                     (rand-nth ["insert" "update"])]
    (cond
        (= dice "insert")
        (let [insert-key   (swap! counter-end inc)]
        (do
            (cassandra/execute
            conn
            (str "BEGIN TRANSACTION "
                "INSERT INTO " keyspace "." table-name
                " (id, balance) values (" insert-key "," amount ");"

                "UPDATE " keyspace "." table-name
                " SET balance = balance - " amount " WHERE id = " from ";"
                "END TRANSACTION;"))
            (assoc op :type :ok :value {:from from, :to insert-key, :amount amount})))

        (= dice "update")
        (do
        (cassandra/execute
        ... 

Jepsen validation log cut:

 {:SI
  {:valid? false,
   :read-count 12015,
   :error-count 5,
   :first-error
   {:type :wrong-total,
    :total 101,
    :op
    {:type :ok,
     :f :read,
     :process 266,
     :time 571865348465,
     :value
     {0 -116,
      1 -333,
      2 -334,

[qvad]

Link to full Jepsen logs https://drive.google.com/file/d/10_RJuDv-es4Yu35ekUVJVpw7LNOlJViY/view?usp=sharing

[qvad]

20220405T111015.000Z.zip
Logs for case w/o any nemesis

[lnguyen-yugabyte]

EDIT:
Changing the statements to

ycqlsh1> BEGIN TRANSACTION INSERT INTO ybdemo_keyspace.bankaccounts (account, balance) VALUES (1,10) IF NOT EXISTS ELSE ERROR; UPDATE ybdemo_keyspace.bankaccounts SET balance = balance - 10 WHERE account = 0;

results in one transaction aborted. The end result state was still correct.

Reproducing on YCQL:

ycqlsh> describe table ybdemo_keyspace.bankaccounts;

CREATE TABLE ybdemo_keyspace.bankaccounts (
    account int PRIMARY KEY,
    balance int
) WITH default_time_to_live = 0
    AND transactions = {'enabled': 'true'};

ycqlsh> insert into ybdemo_keyspace.bankaccounts (account, balance) values (0, 20000);
Now on two ycqlsh shells, execute these:
ycqlsh1> BEGIN TRANSACTION INSERT INTO ybdemo_keyspace.bankaccounts (account, balance) VALUES (1,10); UPDATE ybdemo_keyspace.bankaccounts SET balance = balance - 10 WHERE account = 0;

ycqlsh2> BEGIN TRANSACTION INSERT INTO ybdemo_keyspace.bankaccounts (account, balance) VALUES (1,10); UPDATE ybdemo_keyspace.bankaccounts SET balance = balance - 10 WHERE account = 0;

ycqlsh1> END TRANSACTION;
ycqlsh2> END TRANSACTION;

And voila we hit the error:

ycqlsh> select * from ybdemo_keyspace.bankaccounts;

 account | balance
---------+---------
       1 |      10
       0 |   19980