br: support backup and restore column_stats_usage

[Rustin170506]

What problem does this PR solve?

Issue Number: ref #53567

Problem Summary:

After we enabled the predicate columns feature by default, we need to support backup and restore this information.

What changed and how does it work?

• Added the support for backuping and restoring column_stats_usage.
• I moved some functions from pkg/statistics/handle/usage/predicate_column.go to pkg/statistics/handle/usage/predicatecolumn/predicate_column.go, otherwise it would cause a cyclic dependence issue.

Check List

Tests

• Unit test
• Integration test
• Manual test
• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

支持备份和恢复 mysql.column_stats_usage 表
Support for backing up and restoring mysql.column_stats_usage table

[codecov]

Codecov Report

Attention: Patch coverage is 74.43182% with 45 lines in your changes missing coverage. Please review.

Project coverage is 57.2039%. Comparing base (2108661) to head (5f69217).
Report is 30 commits behind head on master.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #54634         +/-   ##
=================================================
- Coverage   74.6249%   57.2039%   -17.4211%     
=================================================
  Files          1551       1674        +123     
  Lines        362640     617732     +255092     
=================================================
+ Hits         270620     353367      +82747     
- Misses        72390     240802     +168412     
- Partials      19630      23563       +3933     

Flag	Coverage Δ
integration	38.3661% <26.1363%> (?)
unit	72.6265% <74.4318%> (-0.9101%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	52.9656% <ø> (-2.2339%)	⬇️
parser	∅ <ø> (∅)
br	62.7931% <ø> (+15.1614%)	⬆️

[Rustin170506]

/retest

[Rustin170506]

🔢 Self-check (PR reviewed by myself and ready for feedback.)

[Rustin170506]

I am not sure if we have a better way to handle this. The ExecRows function cannot handle nil correctly, so I have to rely on the CONVERT_TZ to insert 'null' if it is 'NULL'.

[Rustin170506]

Tests locally:

1. Start the TiDB cluster with patch: tiup playground nightly --db.binpath /Volumes/t7/code/tidb/bin/tidb-server
2. Build the BR: make build_br
3. Create a table

USE test;

CREATE TABLE new_employees
(
    id INT NOT NULL,
    fname VARCHAR(30),
    lname VARCHAR(30),
    hired DATE NOT NULL DEFAULT '1970-01-01',
    separated DATE DEFAULT '9999-12-31',
    job_code INT,
    store_id INT NOT NULL
);

4. Insert 3 rows

INSERT INTO new_employees (id, fname, lname, hired, separated, job_code, store_id) VALUES
(1, 'John', 'Doe', '2023-01-01', '9999-12-31', 101, 1),
(2, 'Jane', 'Doe', '2023-02-01', '9999-12-31', 102, 2),
(3, 'Jim', 'Beam', '2023-03-01', '9999-12-31', 103, 3);

5. Select the table with column a

SELECT * from mysql.stats_meta;
SELECT * FROM new_employees WHERE  fname = 'a';

6. Wait for dumping the predicate columns into the system table:

SELECT * FROM mysql.column_stats_usage;
+--------+---------+-------------------+----------------+
|table_id|column_id|last_used_at       |last_analyzed_at|
+--------+---------+-------------------+----------------+
|104     |2        |2024-07-17 18:06:39|null            |
+--------+---------+-------------------+----------------+

7. Analyze table: ANALYZE table new_employees; and check the column_stats_usage again:

ANALYZE table new_employees;
SELECT * FROM mysql.column_stats_usage;
+--------+---------+-------------------+-------------------+
|table_id|column_id|last_used_at       |last_analyzed_at   |
+--------+---------+-------------------+-------------------+
|104     |2        |2024-07-17 18:06:39|2024-07-17 18:11:45|
+--------+---------+-------------------+-------------------+

8. Backup the cluster to a local file path: br backup full --pd "127.0.0.1:2379" --storage local:///Volumes/t7/code/brtest --ignore-stats=false

❯ ./br backup full --pd "127.0.0.1:2379" --storage local:///Volumes/t7/code/brtest --ignore-stats=false
Detail BR log in /var/folders/j1/7l6snwpx6svgqxh79bz_d27m0000gn/T/br.log.2024-07-17T18.12.44+0800 
Full Backup <---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%
Checksum <------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%
[2024/07/17 18:12:50.655 +08:00] [INFO] [collector.go:77] ["Full Backup success summary"] [total-ranges=25] [ranges-succeed=25] [ranges-failed=0] [backup-checksum=48.687708ms] [backup-fast-checksum=2.691792ms] [backup-total-ranges=120] [backup-total-regions=120] [total-take=6.39242775s] [backup-data-size(after-compressed)=115.7kB] [Size=115737] [BackupTS=451205179499610114] [total-kv=1433] [total-kv-size=415.9kB] [average-speed=65.06kB/s]

9. Create a new cluster: tiup playground nightly --db.binpath /Volumes/t7/code/tidb/bin/tidb-server
10. Make sure no predicate columns in new cluster at all:

SELECT * FROM mysql.column_stats_usage;

11. Restore from the local file: br restore db --pd "127.0.0.1:2379" --db test --storage local:///Volumes/t7/code/brtest

❯ ./br restore db --pd "127.0.0.1:2379" --db test --storage local:///Volumes/t7/code/brtest                                                                                                   
Detail BR log in /var/folders/j1/7l6snwpx6svgqxh79bz_d27m0000gn/T/br.log.2024-07-17T18.13.53+0800 
DataBase Restore <----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%
[2024/07/17 18:13:57.515 +08:00] [INFO] [collector.go:77] ["DataBase Restore success summary"] [total-ranges=2] [ranges-succeed=2] [ranges-failed=0] [split-region=1.348875ms] [restore-ranges=1] [total-take=3.823280459s] [total-kv=3] [total-kv-size=216B] [average-speed=56.5B/s] [restore-data-size(after-compressed)=1.897kB] [Size=1897] [BackupTS=451205179499610114] [RestoreTS=451205197701316644]

12. Check the column_stats_usage again:

SELECT * FROM mysql.column_stats_usage;
+--------+---------+-------------------+-------------------+
|table_id|column_id|last_used_at       |last_analyzed_at   |
+--------+---------+-------------------+-------------------+
|104     |2        |2024-07-17 18:06:39|2024-07-17 18:11:45|
+--------+---------+-------------------+-------------------+

[Rustin170506]

/retest

[Rustin170506]

/retest

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hawkingrei, Leavrth

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• br/OWNERS [Leavrth]
• pkg/statistics/OWNERS [hawkingrei]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-07-17 11:05:04.312252064 +0000 UTC m=+438326.303193531: ☑️ agreed by hawkingrei.
• 2024-07-19 05:23:35.82006073 +0000 UTC m=+590637.811002201: ☑️ agreed by Leavrth.