statistics: remove statistics.Column.Count

[xuyifangreeneyes]

What problem does this PR solve?

Issue Number: ref #42160 close #44404

Problem Summary:

What is changed and how it works?

Remove statistics.Column.Count.

Before the PR, in order to maintain statistics.Column.Count, we need to read mysql.stats_top_n and mysql.stats_buckets for each column when init stats, which is time-consuming. On the other hand, the usage of statistics.Column.Count is limited. We modify (*DataSource).getColumnNDV to get rid of statistics.Column.Count.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• time-and-fate
• winoros

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[xuyifangreeneyes]

Here the way to calculate column NDV is changed. We assume that the change is ok for most cases.

[xuyifangreeneyes]

If I don't modify the test, I would get the error as follows:

Diff:
--- Expected
+++ Actual
@@ -4,4 +4,4 @@
 [    └─HashAgg 1.00 mpp[tiflash]  funcs:sum(test.partsupp.ps_supplycost)->Column#15]
-[      └─Projection 12500.00 mpp[tiflash]  test.partsupp.ps_supplycost]
-[        └─HashJoin 12500.00 mpp[tiflash]  inner join, equal:[eq(test.supplier.s_suppkey, test.partsupp.ps_suppkey)]]
+[      └─Projection 8000000000.00 mpp[tiflash]  test.partsupp.ps_supplycost]
+[        └─HashJoin 8000000000.00 mpp[tiflash]  inner join, equal:[eq(test.supplier.s_suppkey, test.partsupp.ps_suppkey)]]
 [          ├─ExchangeReceiver(Build) 10000.00 mpp[tiflash]  ]

Before the PR, in getColumnNDV, since hist.Count is 0, we use float64(ds.statisticTable.RealtimeCount) * distinctFactor to calculate NDV. After the PR, if we still analyze partsupp and supplier when they don't have any data, since hist.IsStatsInitialized() is true, we use float64(hist.Histogram.NDV) to calculate NDV. In another word, before the PR we use hist.Count > 0 to decide whether the table is analyzed while after the PR we use hist.IsStatsInitialized() to decide that. I think the latter is more reasonable. I remove the two analyze commands to keep cardinality estimation the same as before. Note that it is rare in the reality that RealtimeCount reaches 800000/10000 but auto analyze is not triggered.

[xuyifangreeneyes]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: c57ea0d

[xuyifangreeneyes]

/test unit-test

[hawkingrei]

/test all

[xuyifangreeneyes]

/test unit-test

[ti-chi-bot]

@xuyifangreeneyes: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
idc-jenkins-ci-tidb/unit-test	6073af3	link	true	/test unit-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[xuyifangreeneyes]

/cherry-pick release-6.5

[ti-chi-bot]

@xuyifangreeneyes: new pull request created to branch release-6.5: #44405.

In response to this:

/cherry-pick release-6.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.