Support user to choose different compression algorithms and compression levels #4161

hehechen · 2022-02-28T10:13:45Z

What problem does this PR solve?

Support user to choose different compression algorithms and compression levels, including lz4, lz4hc, zstd.

Issue Number: close #3671

What is changed and how it works?

Upgrade zstd library to v1.5.1
The original configurations network_compression_method and network_zstd_compression_level are not used in DeltaTree engine. Modify the name of them to dt_compression_method and dt_compression_level, and apply them to DeltaTree engine.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
Add some log in compress and decompress method, modify the configuration values of dt_compression_method and dt_compression_level, check the log to observe whether the compress method and compression level are correct.
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Support zstd compression

ti-chi-bot · 2022-02-28T10:13:47Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

SchrodingerZhu
flowbehappy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

…on levels Signed-off-by: hehechen <awd123456sss@gmail.com>

Signed-off-by: hehechen <awd123456sss@gmail.com>

SchrodingerZhu

The DTTool and DMFile parts LGTM.

hehechen · 2022-03-01T03:19:22Z

/run-all-test

ti-chi-bot · 2022-03-02T02:59:13Z

@hehechen: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-03-02T02:59:18Z

@hehechen: /merge in this pull request requires 2 approval(s).

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sre-bot · 2022-03-02T03:27:18Z

Coverage for changed files

Filename                                                   Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
IO/CompressedWriteBuffer.cpp                                    37                 9    75.68%           3                 0   100.00%          85                 8    90.59%          22                 7    68.18%
IO/CompressionSettings.cpp                                       6                 3    50.00%           2                 0   100.00%          17                 6    64.71%           8                 4    50.00%
Interpreters/Context.cpp                                       471               313    33.55%         163                96    41.10%        1048               678    35.31%         240               183    23.75%
Interpreters/Context.h                                          13                 8    38.46%          13                 8    38.46%          13                 8    38.46%           0                 0         -
Interpreters/Settings.h                                          1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Interpreters/SettingsCommon.h                                  252               200    20.63%         128                90    29.69%         482               386    19.92%          90                82     8.89%
Server/DTTool/DTToolMigrate.cpp                                205                78    61.95%          11                 1    90.91%         282               155    45.04%         124                72    41.94%
Server/tests/gtest_server_config.cpp                          1918               325    83.06%          10                 0   100.00%         395                 3    99.24%         592               293    50.51%
Storages/DeltaMerge/ColumnFile/ColumnFilePersisted.cpp          27                 5    81.48%           8                 0   100.00%          90                11    87.78%          24                 5    79.17%
Storages/DeltaMerge/ColumnFile/ColumnFilePersisted.h             1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Storages/DeltaMerge/ColumnFile/ColumnFileTiny.cpp               70                 4    94.29%          14                 0   100.00%         171                 3    98.25%          40                 4    90.00%
Storages/DeltaMerge/File/DMFileBlockOutputStream.h               5                 1    80.00%           5                 1    80.00%           6                 1    83.33%           0                 0         -
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                         3006               946    68.53%         359               196    45.40%        2591              1259    51.41%        1140               650    42.98%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16667      9477             43.14%    184395  95776        48.06%

full coverage report (for internal network access only)

dbms/src/Interpreters/Settings.h

Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen · 2022-03-02T07:26:40Z

/run-all-tests

sre-bot · 2022-03-02T07:54:43Z

Coverage for changed files

Filename                                                   Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
IO/CompressedWriteBuffer.cpp                                    37                 9    75.68%           3                 0   100.00%          85                 8    90.59%          22                 7    68.18%
IO/CompressionSettings.cpp                                       7                 4    42.86%           2                 0   100.00%          17                 6    64.71%           8                 4    50.00%
Interpreters/Context.cpp                                       471               313    33.55%         163                96    41.10%        1048               678    35.31%         240               183    23.75%
Interpreters/Context.h                                          13                 8    38.46%          13                 8    38.46%          13                 8    38.46%           0                 0         -
Interpreters/Settings.h                                          1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Interpreters/SettingsCommon.h                                  252               200    20.63%         128                90    29.69%         482               386    19.92%          90                82     8.89%
Server/DTTool/DTToolMigrate.cpp                                205                78    61.95%          11                 1    90.91%         281               154    45.20%         124                72    41.94%
Server/tests/gtest_server_config.cpp                          1918               325    83.06%          10                 0   100.00%         395                 3    99.24%         592               293    50.51%
Storages/DeltaMerge/ColumnFile/ColumnFilePersisted.cpp          27                 5    81.48%           8                 0   100.00%          90                11    87.78%          24                 5    79.17%
Storages/DeltaMerge/ColumnFile/ColumnFilePersisted.h             1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Storages/DeltaMerge/ColumnFile/ColumnFileTiny.cpp               70                 4    94.29%          14                 0   100.00%         171                 3    98.25%          40                 4    90.00%
Storages/DeltaMerge/File/DMFileBlockOutputStream.h               5                 1    80.00%           5                 1    80.00%           6                 1    83.33%           0                 0         -
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                         3007               947    68.51%         359               196    45.40%        2590              1258    51.43%        1140               650    42.98%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16667      9478             43.13%    184394  95793        48.05%

full coverage report (for internal network access only)

flowbehappy · 2022-03-02T10:04:47Z

Please also update the doc here: https://docs.pingcap.com/zh/tidb/stable/tiflash-command-line-flags#dttool-migrate

flowbehappy

LGTM

Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen · 2022-03-02T11:18:56Z

/merge

ti-chi-bot · 2022-03-02T11:18:57Z

@hehechen: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-03-02T11:19:02Z

This pull request has been accepted and is ready to merge.

Commit hash: 39a5132

sre-bot · 2022-03-02T11:44:03Z

Coverage for changed files

Filename                                                   Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
IO/CompressedWriteBuffer.cpp                                    37                 9    75.68%           3                 0   100.00%          85                 8    90.59%          22                 7    68.18%
IO/CompressionSettings.cpp                                       9                 7    22.22%           2                 1    50.00%          24                17    29.17%          12                 8    33.33%
Interpreters/Context.cpp                                       471               313    33.55%         163                96    41.10%        1048               678    35.31%         240               183    23.75%
Interpreters/Context.h                                          13                 8    38.46%          13                 8    38.46%          13                 8    38.46%           0                 0         -
Interpreters/Settings.h                                          1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Interpreters/SettingsCommon.h                                  252               200    20.63%         128                90    29.69%         482               386    19.92%          90                82     8.89%
Server/DTTool/DTToolMigrate.cpp                                205                78    61.95%          11                 1    90.91%         281               154    45.20%         124                72    41.94%
Server/tests/gtest_server_config.cpp                          1918               325    83.06%          10                 0   100.00%         395                 3    99.24%         592               293    50.51%
Storages/DeltaMerge/ColumnFile/ColumnFilePersisted.cpp          27                 5    81.48%           8                 0   100.00%          90                11    87.78%          24                 5    79.17%
Storages/DeltaMerge/ColumnFile/ColumnFilePersisted.h             1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Storages/DeltaMerge/ColumnFile/ColumnFileTiny.cpp               70                 4    94.29%          14                 0   100.00%         171                 3    98.25%          40                 4    90.00%
Storages/DeltaMerge/File/DMFileBlockOutputStream.h               5                 1    80.00%           5                 1    80.00%           6                 1    83.33%           0                 0         -
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                         3009               950    68.43%         359               197    45.13%        2597              1269    51.14%        1144               654    42.83%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16689      9455             43.35%    186838  95564        48.85%

full coverage report (for internal network access only)

ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 28, 2022

hehechen requested review from JaySon-Huang and SchrodingerZhu February 28, 2022 10:19

SchrodingerZhu changed the title ~~Compres config~~ Compress config Feb 28, 2022

Support user to choose different compression algorithms and compressi…

bbca60c

…on levels Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen force-pushed the compres_config branch from eb18472 to bbca60c Compare February 28, 2022 10:25

hehechen changed the title ~~Compress config~~ Support user to choose different compression algorithms and compression levels Feb 28, 2022

hehechen requested a review from flowbehappy February 28, 2022 10:29

fix clang tidy

29045dd

Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen force-pushed the compres_config branch from a66bbc9 to 29045dd Compare February 28, 2022 14:49

SchrodingerZhu approved these changes Feb 28, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Feb 28, 2022

flowbehappy reviewed Mar 2, 2022

View reviewed changes

dbms/src/Interpreters/Settings.h Outdated Show resolved Hide resolved

hehechen force-pushed the compres_config branch from 98c20c7 to bb89069 Compare March 2, 2022 06:41

JaySon-Huang reviewed Mar 2, 2022

View reviewed changes

dbms/src/Interpreters/Settings.h Outdated Show resolved Hide resolved

JaySon-Huang reviewed Mar 2, 2022

View reviewed changes

dbms/src/Interpreters/Settings.h Outdated Show resolved Hide resolved

fix comment

edf7c79

Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen force-pushed the compres_config branch from bb89069 to edf7c79 Compare March 2, 2022 07:10

update config template

1914e47

Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen force-pushed the compres_config branch from 300c80c to 1914e47 Compare March 2, 2022 07:20

hehechen requested review from flowbehappy and removed request for JaySon-Huang March 2, 2022 09:04

hehechen requested a review from JaySon-Huang March 2, 2022 09:04

flowbehappy approved these changes Mar 2, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 2, 2022

fix comment

39a5132

Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen force-pushed the compres_config branch from 358a1f6 to 39a5132 Compare March 2, 2022 11:18

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 2, 2022

Merge branch 'master' into compres_config

e7c6548

ti-chi-bot merged commit d321064 into pingcap:master Mar 2, 2022

hehechen deleted the compres_config branch March 2, 2022 14:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support user to choose different compression algorithms and compression levels #4161

Support user to choose different compression algorithms and compression levels #4161

hehechen commented Feb 28, 2022 •

edited

Loading

ti-chi-bot commented Feb 28, 2022 •

edited

Loading

SchrodingerZhu left a comment

hehechen commented Mar 1, 2022

ti-chi-bot commented Mar 2, 2022

ti-chi-bot commented Mar 2, 2022

sre-bot commented Mar 2, 2022

hehechen commented Mar 2, 2022

sre-bot commented Mar 2, 2022

flowbehappy commented Mar 2, 2022 •

edited

Loading

flowbehappy left a comment

hehechen commented Mar 2, 2022

ti-chi-bot commented Mar 2, 2022

ti-chi-bot commented Mar 2, 2022

sre-bot commented Mar 2, 2022

Support user to choose different compression algorithms and compression levels #4161

Support user to choose different compression algorithms and compression levels #4161

Conversation

hehechen commented Feb 28, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Feb 28, 2022 • edited Loading

SchrodingerZhu left a comment

Choose a reason for hiding this comment

hehechen commented Mar 1, 2022

ti-chi-bot commented Mar 2, 2022

ti-chi-bot commented Mar 2, 2022

sre-bot commented Mar 2, 2022

Coverage for changed files

Coverage summary

hehechen commented Mar 2, 2022

sre-bot commented Mar 2, 2022

Coverage for changed files

Coverage summary

flowbehappy commented Mar 2, 2022 • edited Loading

flowbehappy left a comment

Choose a reason for hiding this comment

hehechen commented Mar 2, 2022

ti-chi-bot commented Mar 2, 2022

ti-chi-bot commented Mar 2, 2022

sre-bot commented Mar 2, 2022

Coverage for changed files

Coverage summary

hehechen commented Feb 28, 2022 •

edited

Loading

ti-chi-bot commented Feb 28, 2022 •

edited

Loading

flowbehappy commented Mar 2, 2022 •

edited

Loading