[leo_storage][read_repair] may not satisfy the read consistency

[mocchira]

Problem

The read consisntency(R) may not be satisfied when flowing the exec path through https://github.com/leo-project/leo_storage/blob/1.3.1/src/leo_storage_read_repairer.erl#L89-L90 in case R >= 2 && failed to retrieve the data on erlang:node() for some reason because leo_storage_read_repairer:repair do not check the metadata if a redundancy node is erlang:node().

How to reproduce (The behavior to be is different from the below.)

The most easiest way to reproduce this issue is

• get cluster up with N=2, R=2 and set is_strict_check to true in leo_storage.conf
• put an object
• break the primary object somehow (ex. edit an avs manually and restart a storage node
• get the object
  • expected: 500
  • actual: 200

Solution

Take another look at leo_storage_read_repairer:repair.

• Don't pass a redundancy that already succeeded on leo_storage_handler_object and start (ReadQuorum -1) at https://github.com/leo-project/leo_storage/blob/1.3.1/src/leo_storage_read_repairer.erl#L92 because we are sure that one attempt already succeeded here.
• Check the metadata on every node (currently no check happen on erlang:node()

[Edit]

• Remove strikethroughed sentences to makes it concise.
• Makes sentences that stated what the problem is more explicit

[yosukehara]

@mocchira We need to implement this situation in a LeoFS' CI test. We're going to consider that soon. Thanks.

[mocchira]

@yosukehara updated while leaving(Strikethroughed) the previous wrong comment for the record.
as stated the above, the problem is that leo_storage_read_repairer:repair regard the operation on node() as success regardless of the result of the actual operation.

[mocchira]

works as expected with 88d41ae .
however we also have to fix #543 since metadata is valid but avs is broken in this case.

[yosukehara]

@mocchira I've reviewed 88d41ae , and there is no issue. Thanks.

[yosukehara]

FIXED - https://gist.github.com/yosukehara/f62192c10cb6a43ae3a6f6a976d1ea3a

[yosukehara]

We've checked the read-quorum implementation of v1.3.2:

• Performance evaluation of the read-quorum implementation of v1.3.2, which contains four test cases:
  • R=1, Zero down node
  • R=1, One down node
  • R=2, Zero down node
  • R=2, One down node

After benchmarking that, we've recognized there is no issue.