Enable PageStorage V3 in UT #4884

hehechen · 2022-05-13T02:32:25Z

What problem does this PR solve?

Issue Number: close #4835

What is changed and how it works?

Enable PageStorage V3 in UT and add blob store failpoint test in DeltaMerge Storage UT.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2022-05-13T02:32:26Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

jiaqizho
lidezhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

dbms/src/Storages/DeltaMerge/tests/gtest_dm_delta_merge_store.cpp

dbms/src/Storages/Page/V3/BlobStore.cpp

jiaqizho · 2022-05-16T07:00:17Z

/run-all-tests

jiaqizho · 2022-05-16T07:58:25Z

[2022-05-16T07:52:04.474Z]    12876 ms: /tiflash/gtests_dbms DatabaseTiFlashTest.RenameTable
[2022-05-16T07:52:04.474Z]     1681 ms: /tiflash/gtests_dbms DatabaseTiFlashTest.RenameTableBetweenDatabase
[2022-05-16T07:52:04.474Z]     1578 ms: /tiflash/gtests_dbms UsersConfigParserTest.ReloadStoragePoolConfig
[2022-05-16T07:52:04.474Z]     1802 ms: /tiflash/gtests_dbms StorageDeltaMergeTest.ReadWriteCase1
[2022-05-16T07:52:04.474Z]     1728 ms: /tiflash/gtests_dbms DMMinMaxIndexTest.NullableToNullable
[2022-05-16T07:52:04.474Z]     1734 ms: /tiflash/gtests_dbms StorageDeltaMergeTest.Rename
[2022-05-16T07:52:04.474Z]     1631 ms: /tiflash/gtests_dbms DMMinMaxIndexTest.Logical
[2022-05-16T07:52:04.474Z]     1751 ms: /tiflash/gtests_dbms StorageDeltaMergeTest.HandleCol
[2022-05-16T07:52:04.474Z]     1699 ms: /tiflash/gtests_dbms DMMinMaxIndexTest.DelMark
[2022-05-16T07:52:04.474Z]     1765 ms: /tiflash/gtests_dbms StorageDeltaMergeTest.RestoreAfterClearData
[2022-05-16T07:52:04.474Z]     2675 ms: /tiflash/gtests_dbms RegionPersister_test.persister
[2022-05-16T07:52:04.474Z]     1585 ms: /tiflash/gtests_dbms RegionPersister_test.persister_compatible_mode
[2022-05-16T07:52:04.474Z]     1689 ms: /tiflash/gtests_dbms TestMode/DeltaMergeStoreRWTest.WriteCrashBeforeWalWithCache/V1_BlockOnly
[2022-05-16T07:52:04.474Z]     1822 ms: /tiflash/gtests_dbms TestMode/DeltaMergeStoreRWTest.WriteCrashBeforeWalWithCache/V2_BlockOnly
[2022-05-16T07:52:04.474Z]     1693 ms: /tiflash/gtests_dbms TestMode/DeltaMergeStoreRWTest.WriteCrashBeforeWalWithCache/V2_FileOnly
[2022-05-16T07:52:04.474Z]     1634 ms: /tiflash/gtests_dbms TestMode/DeltaMergeStoreRWTest.WriteCrashBeforeWalWithCache/V2_Mix

There are still some tests that are not compatible with PageFormat::V3.

Could you fix that and found out whether any errors exist in V3?

hehechen · 2022-05-16T09:59:19Z

/run-unit-test

hehechen · 2022-05-17T15:25:47Z

/run-unit-test

hehechen · 2022-05-19T10:52:30Z

/run-unit-test

hehechen · 2022-05-20T09:50:45Z

/run-unit-test

jiaqizho · 2022-05-20T12:53:56Z

Could you test #4948?

hehechen · 2022-05-23T03:42:54Z

Could you test #4948?

#4948 has passed all tests.

jiaqizho · 2022-05-24T03:15:31Z

/run-all-tests

Signed-off-by: hehechen <awd123456sss@gmail.com>

jiaqizho · 2022-05-24T03:29:01Z

@hehechen any update/missing in this PR?

hehechen · 2022-05-24T03:33:13Z

@hehechen any update/missing in this PR?

fix clang-tidy and some failed UT after enable PageStorage V3.

Signed-off-by: hehechen <awd123456sss@gmail.com>

jiaqizho · 2022-05-24T05:30:38Z

/run-all-tests

Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen · 2022-05-24T07:12:52Z

/run-all-tests

hehechen · 2022-05-24T08:01:00Z

/run-unit-test

sre-bot · 2022-05-24T08:13:58Z

Coverage for changed files

Filename                                                     Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Interpreters/Context.cpp                                         512               323    36.91%         166                88    46.99%        1126               693    38.45%         272               205    24.63%
Server/tests/gtest_server_config.cpp                            2186               351    83.94%          10                 0   100.00%         423                 3    99.29%         676               335    50.44%
Storages/DeltaMerge/StoragePool.cpp                              224               110    50.89%          31                 9    70.97%         491               219    55.40%         146                82    43.84%
Storages/DeltaMerge/tests/gtest_dm_delta_merge_store.cpp        6851              1829    73.30%          47                 1    97.87%        3059               103    96.63%        2262              1013    55.22%
Storages/Page/V3/tests/gtest_page_storage_mix_mode.cpp          1504               404    73.14%          14                 0   100.00%         406                21    94.83%         442               226    48.87%
Storages/Transaction/tests/gtest_region_persister.cpp           1010               328    67.52%          16                 3    81.25%         388               117    69.85%         284               148    47.89%
TestUtils/TiFlashTestEnv.cpp                                      25                 4    84.00%           4                 0   100.00%          65                 9    86.15%          14                 6    57.14%
TestUtils/TiFlashTestEnv.h                                        25                 3    88.00%           6                 0   100.00%          46                 5    89.13%          16                 3    81.25%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                          12337              3352    72.83%         294               101    65.65%        6004              1170    80.51%        4112              2018    50.92%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18224      9787             46.30%    204290  97954        52.05%

full coverage report (for internal network access only)

dbms/src/Storages/DeltaMerge/tests/gtest_dm_delta_merge_store.cpp

dbms/src/Server/tests/gtest_server_config.cpp

dbms/src/Storages/Transaction/tests/gtest_region_persister.cpp

jiaqizho

LGTM with some comments

Signed-off-by: hehechen <awd123456sss@gmail.com>

dbms/src/Interpreters/Context.cpp

hehechen · 2022-05-25T02:10:56Z

/run-all-tests

sre-bot · 2022-05-25T02:35:10Z

Coverage for changed files

Filename                                                     Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Interpreters/Context.cpp                                         512               323    36.91%         166                88    46.99%        1126               693    38.45%         272               205    24.63%
Server/tests/gtest_server_config.cpp                            2186               351    83.94%          10                 0   100.00%         423                 3    99.29%         676               335    50.44%
Storages/DeltaMerge/StoragePool.cpp                              224               110    50.89%          31                 9    70.97%         491               219    55.40%         146                82    43.84%
Storages/DeltaMerge/tests/gtest_dm_delta_merge_store.cpp        6851              1829    73.30%          47                 1    97.87%        3059               103    96.63%        2262              1013    55.22%
Storages/Page/PageStorage.h                                       19                 2    89.47%          19                 2    89.47%          85                 4    95.29%           0                 0         -
Storages/Page/V3/tests/gtest_page_storage_mix_mode.cpp          1504               404    73.14%          14                 0   100.00%         406                21    94.83%         442               226    48.87%
Storages/Transaction/tests/gtest_region_persister.cpp           1011               328    67.56%          17                 3    82.35%         391               117    70.08%         284               148    47.89%
TestUtils/TiFlashTestEnv.cpp                                      25                 4    84.00%           4                 0   100.00%          65                 9    86.15%          14                 6    57.14%
TestUtils/TiFlashTestEnv.h                                        25                 3    88.00%           6                 0   100.00%          46                 5    89.13%          16                 3    81.25%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                          12357              3354    72.86%         314               103    67.20%        6092              1174    80.73%        4112              2018    50.92%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18225      9787             46.30%    204293  97941        52.06%

full coverage report (for internal network access only)

hehechen · 2022-05-25T06:54:59Z

/merge

ti-chi-bot · 2022-05-25T06:55:00Z

@hehechen: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-05-25T06:55:02Z

This pull request has been accepted and is ready to merge.

Commit hash: 962dad0

sre-bot · 2022-05-25T07:33:20Z

Coverage for changed files

Filename                                                     Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Interpreters/Context.cpp                                         512               323    36.91%         166                88    46.99%        1126               693    38.45%         272               205    24.63%
Server/tests/gtest_server_config.cpp                            2186               351    83.94%          10                 0   100.00%         423                 3    99.29%         676               335    50.44%
Storages/DeltaMerge/tests/gtest_dm_delta_merge_store.cpp        6851              1829    73.30%          47                 1    97.87%        3059               103    96.63%        2262              1013    55.22%
Storages/Page/PageStorage.h                                       19                 2    89.47%          19                 2    89.47%          85                 4    95.29%           0                 0         -
Storages/Page/V3/tests/gtest_page_storage_mix_mode.cpp          1504               404    73.14%          14                 0   100.00%         406                21    94.83%         442               226    48.87%
Storages/Transaction/tests/gtest_region_persister.cpp           1011               328    67.56%          17                 3    82.35%         391               117    70.08%         284               148    47.89%
TestUtils/TiFlashTestEnv.cpp                                      25                 4    84.00%           4                 0   100.00%          65                 9    86.15%          14                 6    57.14%
TestUtils/TiFlashTestEnv.h                                        25                 3    88.00%           6                 0   100.00%          46                 5    89.13%          16                 3    81.25%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                          12133              3244    73.26%         283                94    66.78%        5601               955    82.95%        3966              1936    51.19%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18256      9788             46.38%    204413  97994        52.06%

full coverage report (for internal network access only)

This reverts commit 0f56b15.

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 13, 2022

hehechen force-pushed the pagestorage_test branch from f14952e to 93addf7 Compare May 13, 2022 07:13

hehechen requested review from JaySon-Huang, jiaqizho and lidezhu May 13, 2022 07:16

lidezhu reviewed May 13, 2022

View reviewed changes

jiaqizho reviewed May 13, 2022

View reviewed changes

dbms/src/Storages/Page/V3/BlobStore.cpp Outdated Show resolved Hide resolved

hehechen force-pushed the pagestorage_test branch from 5ed3c01 to f81ea72 Compare May 16, 2022 02:42

hehechen requested review from jiaqizho and lidezhu May 16, 2022 02:42

hehechen changed the title ~~Enable PageStorage V3 in UT and add blobstore failpoint~~ Enable PageStorage V3 in UT May 16, 2022

hehechen force-pushed the pagestorage_test branch from 92d3e22 to 15651ff Compare May 19, 2022 10:52

hehechen force-pushed the pagestorage_test branch from 15651ff to 595da61 Compare May 19, 2022 15:22

This was referenced May 19, 2022

PageStorage V3 GC can't work when restore #4937

Closed

PageStorage V3 GC exception #4939

Closed

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 24, 2022

Enable PageStorage V3 in UT

6d2fcf1

Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen force-pushed the pagestorage_test branch from 434a9da to 6d2fcf1 Compare May 24, 2022 03:22

ti-chi-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 24, 2022

fix code

59b3958

Signed-off-by: hehechen <awd123456sss@gmail.com>

fix ut

1fa80da

Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen force-pushed the pagestorage_test branch from 8badf65 to 1fa80da Compare May 24, 2022 08:30

jiaqizho reviewed May 24, 2022

View reviewed changes

jiaqizho approved these changes May 24, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 24, 2022

fix comment

962dad0

Signed-off-by: hehechen <awd123456sss@gmail.com>

JaySon-Huang reviewed May 24, 2022

View reviewed changes

dbms/src/Interpreters/Context.cpp Show resolved Hide resolved

lidezhu approved these changes May 25, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 25, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 25, 2022

ti-chi-bot added 2 commits May 25, 2022 14:55

Merge branch 'master' into pagestorage_test

f7c172b

Merge branch 'master' into pagestorage_test

d925f89

ti-chi-bot merged commit 0f56b15 into pingcap:master May 25, 2022

hehechen added a commit that referenced this pull request May 27, 2022

Revert "Enable PageStorage V3 in UT (#4884)"

2ae7366

This reverts commit 0f56b15.

hehechen mentioned this pull request May 27, 2022

Revert "Enable PageStorage V3 in UT" #5022

Closed

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable PageStorage V3 in UT #4884

Enable PageStorage V3 in UT #4884

hehechen commented May 13, 2022 •

edited

Loading

ti-chi-bot commented May 13, 2022 •

edited

Loading

jiaqizho commented May 16, 2022

jiaqizho commented May 16, 2022 •

edited

Loading

hehechen commented May 16, 2022

hehechen commented May 17, 2022

hehechen commented May 19, 2022

hehechen commented May 20, 2022

jiaqizho commented May 20, 2022

hehechen commented May 23, 2022

jiaqizho commented May 24, 2022

jiaqizho commented May 24, 2022

hehechen commented May 24, 2022

jiaqizho commented May 24, 2022

hehechen commented May 24, 2022

hehechen commented May 24, 2022

sre-bot commented May 24, 2022

jiaqizho left a comment

hehechen commented May 25, 2022

sre-bot commented May 25, 2022

hehechen commented May 25, 2022

ti-chi-bot commented May 25, 2022

ti-chi-bot commented May 25, 2022

sre-bot commented May 25, 2022

Enable PageStorage V3 in UT #4884

Enable PageStorage V3 in UT #4884

Conversation

hehechen commented May 13, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented May 13, 2022 • edited Loading

jiaqizho commented May 16, 2022

jiaqizho commented May 16, 2022 • edited Loading

hehechen commented May 16, 2022

hehechen commented May 17, 2022

hehechen commented May 19, 2022

hehechen commented May 20, 2022

jiaqizho commented May 20, 2022

hehechen commented May 23, 2022

jiaqizho commented May 24, 2022

jiaqizho commented May 24, 2022

hehechen commented May 24, 2022

jiaqizho commented May 24, 2022

hehechen commented May 24, 2022

hehechen commented May 24, 2022

sre-bot commented May 24, 2022

Coverage for changed files

Coverage summary

jiaqizho left a comment

Choose a reason for hiding this comment

hehechen commented May 25, 2022

sre-bot commented May 25, 2022

Coverage for changed files

Coverage summary

hehechen commented May 25, 2022

ti-chi-bot commented May 25, 2022

ti-chi-bot commented May 25, 2022

sre-bot commented May 25, 2022

Coverage for changed files

Coverage summary

hehechen commented May 13, 2022 •

edited

Loading

ti-chi-bot commented May 13, 2022 •

edited

Loading

jiaqizho commented May 16, 2022 •

edited

Loading