*: fix the estimation error on normal column when collation enabled (#18104)

[ti-srebot]

cherry-pick #18104 to release-4.0

---

What problem does this PR solve?

Issue Number: close #14689

Problem Summary:

For index, its key is already generated by the sort key by the collation information. And when we query the index estimation, we also use EncodeKey which will first convert the column value to sort key then encode it. So it's automatically correct for index without any additional change.

But for a column, when sampling we use its original value thus when query it with count-min sketch or histogram. We'll get a wrong answer since the order information is lost.

What is changed and how it works?

What's Changed:

For tikv part, use the sort key generated by the collation as the sampling data. tikv/tikv#8105
And when querying, convert to sort key first.

How it Works:

Related changes

• Need to cherry-pick to the release branch

Check List

Tests

• Unit test -> Unit test is removed. mock-tikv doesn't support the new row format. It's a little hard to add test for it. But unistore is not included in 4.0
• Integration test

Side effects

• Performance regression
  • Consumes more CPU

Release note

• Fix the row count estimation error for a non-index column with collation enabled.

[ti-srebot]

/run-all-tests

[ti-srebot]

@winoros please accept the invitation then you can push to the cherry-pick pull requests.
https://github.com/ti-srebot/tidb/invitations

[qw4990]

There are many unnecessary changes in this PR, please fix them @winoros

[zz-jason]

@winoros please resolve conflicts and fix CI.

[wjhuang2016]

LGTM

[lzmhhh123]

LGTM

[zz-jason]

LGTM

[zz-jason]

/merge

[ti-srebot]

Your auto merge job has been accepted, waiting for:

• 19834
• 19835

[ti-srebot]

/run-all-tests

[ti-srebot]

@ti-srebot merge failed.

[winoros]

Tested with tikv/tikv#8620

MySQL [test]> create table t(a varchar(20) collate utf8mb4_general_ci);
Query OK, 0 rows affected (0.10 sec)

MySQL [test]> insert into t values('aaa'), ('bbb'), ('AAA'), ('BBB');
Query OK, 4 rows affected (0.01 sec)
Records: 4  Duplicates: 0  Warnings: 0

MySQL [test]> analyze table t;
Query OK, 0 rows affected (0.05 sec)

MySQL [test]> explain select * from t where a='aÄa';
+-------------------------+---------+-----------+---------------+----------------------+
| id                      | estRows | task      | access object | operator info        |
+-------------------------+---------+-----------+---------------+----------------------+
| TableReader_7           | 2.00    | root      |               | data:Selection_6     |
| └─Selection_6           | 2.00    | cop[tikv] |               | eq(test.t.a, "aÄa")  |
|   └─TableFullScan_5     | 4.00    | cop[tikv] | table:t       | keep order:false     |
+-------------------------+---------+-----------+---------------+----------------------+
3 rows in set (0.00 sec)

MySQL [test]> show stats_buckets;
+---------+------------+----------------+-------------+----------+-----------+-------+---------+-------------+-------------+
| Db_name | Table_name | Partition_name | Column_name | Is_index | Bucket_id | Count | Repeats | Lower_Bound | Upper_Bound |
+---------+------------+----------------+-------------+----------+-----------+-------+---------+-------------+-------------+
| test    | t          |                | a           |        0 |         0 |     2 |       2 |  A A A      |  A A A      |
| test    | t          |                | a           |        0 |         1 |     4 |       2 |  B B B      |  B B B      |
+---------+------------+----------------+-------------+----------+-----------+-------+---------+-------------+-------------+
2 rows in set (0.00 sec)

[winoros]

/run-all-tests

[winoros]

It's safe to merge this one before the related tikv pr.

[AilinKid]

Is it a 4.0.6 release blocker?

[winoros]

@AilinKid The tikv side will be merged. So this one should be merged too.