(*batchTx) UnsafeDelete may break the linearizability

[ahrtr]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

TL; DR

(*batchTx) UnsafeDelete may break the linearizability. etcd may still read stale data when during applying a request if previous applying process includes (*batchTx) UnsafeDelete request(s).

Root cause

Please refer to

• Commit bbolt transaction if there is any pending deleting operations #17157
• Commit transaction for each configuration change #17119 (comment)
• Test case TestRangeAfterDeleteMatch in Reproduce readTxn breaks read commited on delete #17124

Solution

The PR should contains the following content/commits,

• Add a couple of new test case on removing users/roles to reproduce the issue
  • Let's use the auth users as an example. Firstly add an user, then remove it and immediately after add it again. The client may get an ErrUserAlreadyExist error.
• Cherry-pick the test case TestRangeAfterDeleteMatch in Reproduce readTxn breaks read commited on delete #17124;
• Cherry-pick Commit bbolt transaction if there is any pending deleting operations #17157: (it is the simplest solution I can think of). Confirm that it can fix the test failures
  • Please fix the test failures
• Please feel free to suggest any other test case

Backport

The fix should be backported to both 3.5 and. 3.4

What did you expect to happen?

Ensure linearizability on any deletion operations.

How can we reproduce it (as minimally and precisely as possible)?

See above

Anything else we need to know?

No response

Etcd version (please run commands below)

all releases

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

[serathius]

We are seeing more and more compounding issues with batchTxBuffered, I think we should consider two things:

• More generic testing like proposed in Revise backend transaction testing #17126, there is just too much risk of missing an issue due to how complicated the code is.
• Better impact analysis. We know that txn correctness issues don't affect KV store or they would be found by robustness tests. However there is large API surfaced not covered by robustness tests. Things like authorization, membership, leases etc. I think we need to add a verifier that will confirm correct behavior of backend. See https://github.com/etcd-io/etcd/blob/main/server/verify/verify.go#L46 as an example