Add `Precompiled header` for modules & Refine by PImpl to accelerate release build #5047

solotzg · 2022-06-02T03:13:33Z

What problem does this PR solve?

Issue Number: ref #4909

Problem Summary:

Ref: https://solotzg.github.io/2022/06/01/system/cpp-compile-optimize/#%E9%A2%84%E7%BC%96%E8%AF%91%E5%A4%B4%E6%96%87%E4%BB%B6%EF%BC%88PCH%EF%BC%89

Before

**** Time summary:
Compilation (5485 times):
  Parsing (frontend):         7181.8 s
  Codegen & opts (backend):   6224.5 s

*** Expensive headers:
1091837 ms: /root/work/tiflash/dbms/src/Common/FmtUtils.h (included 750 times, avg 1455 ms), included via:
1022467 ms: /root/work/tiflash/libs/libcommon/include/common/StringRef.h (included 754 times, avg 1356 ms), included via:
985876 ms: /root/work/tiflash/dbms/src/Common/Exception.h (included 748 times, avg 1318 ms), included via:
974680 ms: /root/work/tiflash/dbms/src/Common/StackTrace.h (included 749 times, avg 1301 ms), included via:
681167 ms: /data2/tiflash-env/sysroot/include/c++/v1/string (included 1903 times, avg 357 ms), included via:
501559 ms: /data2/tiflash-env/sysroot/include/c++/v1/algorithm (included 1969 times, avg 254 ms), included via:
498058 ms: /root/work/tiflash/dbms/src/Columns/IColumn.h (included 516 times, avg 965 ms), included via:
422329 ms: /root/work/tiflash/dbms/src/Core/Types.h (included 706 times, avg 598 ms), included via:
405688 ms: /data2/tiflash-env/sysroot/include/c++/v1/functional (included 1971 times, avg 205 ms), included via:

After

**** Time summary:
Compilation (5488 times):
  Parsing (frontend):         4208.2 s
  Codegen & opts (backend):   6663.9 s

*** Expensive headers:
188325 ms: /data2/work/tiflash/dbms/src/Columns/IColumn.h (included 515 times, avg 365 ms), included via:
...
177991 ms: /data2/work/tiflash/dbms/src/Common/Decimal.h (included 657 times, avg 270 ms), included via:
...
122434 ms: /data2/work/tiflash/dbms/src/Core/Block.h (included 415 times, avg 295 ms), included via:
...
115633 ms: /data2/work/tiflash/dbms/src/Core/Field.h (included 656 times, avg 176 ms), included via:
...

With the help of LTO(#4890, #4924), Forward declaration won't hurt performance because linker(lld) will make short functions inline automatically.

For example: use objdump -d, there will be only one entry to function PlusDecimalInferer::infer in code section. There is no other function call towards PlusDecimalInferer::infer by instruction callq.

➜  release-centos7-llvm git:(pch) ✗ ./tiflash/tiflash version
TiFlash
Release Version: v6.2.0-alpha-30-gf2a406a
Edition:         Community
Git Commit Hash: f2a406aa93f06e98e1868597f0ff25ae87ed40c5
Git Branch:      pch
UTC Build Time:  2022-06-05 07:30:43
Enable Features: jemalloc avx avx512 unwind thinlto
Profile:         RELWITHDEBINFO

Raft Proxy
Git Commit Hash:   ca2f51f94e55bdd23749dcc02ab4afb94eeb5ae5
Git Commit Branch: HEAD
UTC Build Time:    2022-06-05 07:33:25
Rust Version:      rustc 1.60.0-nightly (1e12aef3f 2022-02-13)
Storage Engine:    tiflash
Prometheus Prefix: tiflash_proxy_
Profile:           release

➜  release-centos7-llvm git:(pch) ✗ objdump -d ./tiflash/tiflash | grep 'PlusDecimalInferer'
0000000001ce43c0 <_ZN2DB18PlusDecimalInferer5inferEjjjj>:```

➜  release-centos7-llvm git:(pch) ✗ objdump -d ./tiflash/tiflash | grep '1ce43c0'           
0000000001ce43c0 <_ZN2DB18PlusDecimalInferer5inferEjjjj>:
 1ce43c0:       55                      push   %rbp

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2022-06-02T03:13:34Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

SchrodingerZhu
fuzhe1989

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

solotzg · 2022-06-02T03:20:30Z

/hold

solotzg · 2022-06-02T05:38:58Z

/run-unit-test

solotzg · 2022-06-02T06:00:23Z

/run-integration-test

solotzg · 2022-06-07T04:49:37Z

/run-all-tests

sre-bot · 2022-06-07T05:20:26Z

Coverage for changed files

Filename                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/FmtUtils.h                                        17                 2    88.24%          11                 2    81.82%          39                 2    94.87%           4                 0   100.00%
Common/TiFlashException.cpp                              98                98     0.00%          13                13     0.00%          95                95     0.00%          34                34     0.00%
Common/TiFlashException.h                                 8                 4    50.00%           6                 4    33.33%           6                 4    33.33%           4                 2    50.00%
DataStreams/TiRemoteBlockInputStream.h                  117               117     0.00%          17                17     0.00%         164               164     0.00%          60                60     0.00%
Flash/Coprocessor/CoprocessorReader.h                    49                49     0.00%          11                11     0.00%          80                80     0.00%          32                32     0.00%
Flash/Coprocessor/DAGStorageInterpreter.cpp             424               424     0.00%          32                32     0.00%         870               870     0.00%         282               282     0.00%
Flash/Coprocessor/GenSchemaAndColumn.cpp                 10                 2    80.00%           2                 0   100.00%          32                 6    81.25%          10                 3    70.00%
Flash/Coprocessor/RemoteRequest.cpp                      31                31     0.00%           3                 3     0.00%          80                80     0.00%          22                22     0.00%
Flash/Coprocessor/RemoteRequest.h                         1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
Flash/Coprocessor/TablesRegionsInfo.cpp                  41                41     0.00%           5                 5     0.00%          62                62     0.00%          26                26     0.00%
Flash/Coprocessor/TablesRegionsInfo.h                    14                13     7.14%           8                 7    12.50%          27                26     3.70%           6                 6     0.00%
Flash/Management/tests/gtest_manual_compact.cpp         905               315    65.19%          16                 0   100.00%         251                 4    98.41%         264               141    46.59%
Flash/Mpp/MinTSOScheduler.h                               4                 1    75.00%           2                 0   100.00%           4                 0   100.00%           4                 3    25.00%
Server/CLIService.h                                      37                37     0.00%           7                 7     0.00%         109               109     0.00%          24                24     0.00%
Storages/Transaction/TMTContext.h                         3                 2    33.33%           3                 2    33.33%           3                 2    33.33%           0                 0         -
TestUtils/ColumnsToTiPBExpr.cpp                          55                 5    90.91%           6                 0   100.00%         122                11    90.98%          40                 7    82.50%
TiDB/Schema/TiDBSchemaSyncer.h                          140               132     5.71%          13                 9    30.77%         125               100    20.00%          52                51     1.92%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                  1954              1274    34.80%         156               113    27.56%        2070              1616    21.93%         864               693    19.79%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18284      9733             46.77%    205116  97578        52.43%

full coverage report (for internal network access only)

solotzg · 2022-06-07T15:57:55Z

/merge

ti-chi-bot · 2022-06-07T15:57:57Z

@solotzg: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-06-07T15:57:58Z

This pull request has been accepted and is ready to merge.

Commit hash: 5662fdd

solotzg · 2022-06-07T16:00:33Z

/hold

sre-bot · 2022-06-07T16:30:07Z

Coverage for changed files

Filename                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/FmtUtils.h                                        17                 2    88.24%          11                 2    81.82%          39                 2    94.87%           4                 0   100.00%
Common/TiFlashException.cpp                              98                98     0.00%          13                13     0.00%          95                95     0.00%          34                34     0.00%
Common/TiFlashException.h                                 8                 4    50.00%           6                 4    33.33%           6                 4    33.33%           4                 2    50.00%
DataStreams/TiRemoteBlockInputStream.h                  117               117     0.00%          17                17     0.00%         164               164     0.00%          60                60     0.00%
Flash/Coprocessor/CoprocessorReader.h                    49                49     0.00%          11                11     0.00%          80                80     0.00%          32                32     0.00%
Flash/Coprocessor/DAGStorageInterpreter.cpp             424               424     0.00%          32                32     0.00%         870               870     0.00%         282               282     0.00%
Flash/Coprocessor/GenSchemaAndColumn.cpp                 10                 2    80.00%           2                 0   100.00%          32                 6    81.25%          10                 3    70.00%
Flash/Coprocessor/RemoteRequest.cpp                      31                31     0.00%           3                 3     0.00%          80                80     0.00%          22                22     0.00%
Flash/Coprocessor/RemoteRequest.h                         1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
Flash/Coprocessor/TablesRegionsInfo.cpp                  41                41     0.00%           5                 5     0.00%          62                62     0.00%          26                26     0.00%
Flash/Coprocessor/TablesRegionsInfo.h                    14                13     7.14%           8                 7    12.50%          27                26     3.70%           6                 6     0.00%
Flash/Management/tests/gtest_manual_compact.cpp         905               315    65.19%          16                 0   100.00%         251                 4    98.41%         264               141    46.59%
Flash/Mpp/MinTSOScheduler.h                               4                 1    75.00%           2                 0   100.00%           4                 0   100.00%           4                 3    25.00%
Interpreters/Context.h                                   13                 5    61.54%          13                 5    61.54%          13                 5    61.54%           0                 0         -
Server/CLIService.h                                      37                37     0.00%           7                 7     0.00%         109               109     0.00%          24                24     0.00%
Storages/Transaction/KVStore.cpp                        387                76    80.36%          41                 3    92.68%         675                79    88.30%         208                75    63.94%
Storages/Transaction/KVStore.h                            5                 3    40.00%           5                 3    40.00%           5                 3    40.00%           0                 0         -
Storages/Transaction/ProxyFFI.cpp                       174               110    36.78%          41                24    41.46%         346               235    32.08%          68                31    54.41%
Storages/Transaction/ReadIndexWorker.cpp                299                 8    97.32%          80                 3    96.25%         693                 9    98.70%         170                10    94.12%
Storages/Transaction/TMTContext.h                         3                 2    33.33%           3                 2    33.33%           3                 2    33.33%           0                 0         -
TestUtils/ColumnsToTiPBExpr.cpp                          55                 5    90.91%           6                 0   100.00%         122                11    90.98%          40                 7    82.50%
TiDB/Schema/TiDBSchemaSyncer.h                          140               132     5.71%          13                 9    30.77%         125               100    20.00%          52                51     1.92%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                  2832              1476    47.88%         336               151    55.06%        3802              1947    48.79%        1310               809    38.24%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18284      9733             46.77%    205113  97577        52.43%

full coverage report (for internal network access only)

solotzg · 2022-06-07T16:34:26Z

/merge

ti-chi-bot · 2022-06-07T16:34:27Z

@solotzg: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-06-07T16:34:29Z

This pull request has been accepted and is ready to merge.

Commit hash: 96567cd

sre-bot · 2022-06-07T17:02:48Z

Coverage for changed files

Filename                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/FmtUtils.h                                        17                 2    88.24%          11                 2    81.82%          39                 2    94.87%           4                 0   100.00%
Common/TiFlashException.cpp                              98                98     0.00%          13                13     0.00%          95                95     0.00%          34                34     0.00%
Common/TiFlashException.h                                 8                 4    50.00%           6                 4    33.33%           6                 4    33.33%           4                 2    50.00%
DataStreams/TiRemoteBlockInputStream.h                  117               117     0.00%          17                17     0.00%         164               164     0.00%          60                60     0.00%
Flash/Coprocessor/CoprocessorReader.h                    49                49     0.00%          11                11     0.00%          80                80     0.00%          32                32     0.00%
Flash/Coprocessor/DAGStorageInterpreter.cpp             424               424     0.00%          32                32     0.00%         870               870     0.00%         282               282     0.00%
Flash/Coprocessor/GenSchemaAndColumn.cpp                 10                 2    80.00%           2                 0   100.00%          32                 6    81.25%          10                 3    70.00%
Flash/Coprocessor/RemoteRequest.cpp                      31                31     0.00%           3                 3     0.00%          80                80     0.00%          22                22     0.00%
Flash/Coprocessor/RemoteRequest.h                         1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
Flash/Coprocessor/TablesRegionsInfo.cpp                  41                41     0.00%           5                 5     0.00%          62                62     0.00%          26                26     0.00%
Flash/Coprocessor/TablesRegionsInfo.h                    14                13     7.14%           8                 7    12.50%          27                26     3.70%           6                 6     0.00%
Flash/Management/tests/gtest_manual_compact.cpp         905               315    65.19%          16                 0   100.00%         251                 4    98.41%         264               141    46.59%
Flash/Mpp/MinTSOScheduler.h                               4                 1    75.00%           2                 0   100.00%           4                 0   100.00%           4                 3    25.00%
Interpreters/Context.h                                   13                 5    61.54%          13                 5    61.54%          13                 5    61.54%           0                 0         -
Server/CLIService.h                                      37                37     0.00%           7                 7     0.00%         109               109     0.00%          24                24     0.00%
Storages/Transaction/KVStore.cpp                        387                77    80.10%          41                 3    92.68%         675                79    88.30%         208                77    62.98%
Storages/Transaction/KVStore.h                            5                 3    40.00%           5                 3    40.00%           5                 3    40.00%           0                 0         -
Storages/Transaction/ProxyFFI.cpp                       174               110    36.78%          41                24    41.46%         346               235    32.08%          68                31    54.41%
Storages/Transaction/ReadIndexWorker.cpp                299                 8    97.32%          80                 3    96.25%         693                 9    98.70%         170                11    93.53%
Storages/Transaction/TMTContext.h                         3                 2    33.33%           3                 2    33.33%           3                 2    33.33%           0                 0         -
TestUtils/ColumnsToTiPBExpr.cpp                          55                 5    90.91%           6                 0   100.00%         122                11    90.98%          40                 7    82.50%
TiDB/Schema/TiDBSchemaSyncer.h                          140               132     5.71%          13                 9    30.77%         125               100    20.00%          52                51     1.92%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                  2832              1477    47.85%         336               151    55.06%        3802              1947    48.79%        1310               812    38.02%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18284      9733             46.77%    205113  97578        52.43%

full coverage report (for internal network access only)

solotzg · 2022-06-08T01:05:47Z

/unhold

solotzg added 7 commits May 26, 2022 12:19

Optimize modules by PCH and PImpl

70fb05e

wip

6858eed

optimize modules

139a3df

add comment

f39e8b1

fix pch options

217c140

wip

3da1c11

Merge remote-tracking branch 'pingcap/master' into pch

f9f5da5

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 2, 2022

solotzg added type/enhancement The issue or PR belongs to an enhancement. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 2, 2022

solotzg mentioned this pull request Jun 2, 2022

Enhancement For Code Quality & Performance #4909

Open

6 tasks

solotzg requested review from fuzhe1989, zanmato1984 and SchrodingerZhu and removed request for zanmato1984, fuzhe1989 and SchrodingerZhu June 2, 2022 03:17

ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 2, 2022

solotzg closed this Jun 2, 2022

solotzg reopened this Jun 2, 2022

ti-chi-bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jun 2, 2022

solotzg force-pushed the pch branch from 61b4919 to fefc5e3 Compare June 2, 2022 03:45

Fix compile

f5324f7

solotzg force-pushed the pch branch from fefc5e3 to f5324f7 Compare June 2, 2022 05:03

Fix compile2

d532a7f

solotzg requested a review from fuzhe1989 June 6, 2022 12:56

rollback modification

dae201b

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 7, 2022

Merge remote-tracking branch 'pingcap/master' into pch

665f621

fuzhe1989 approved these changes Jun 7, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 7, 2022

solotzg added 2 commits June 7, 2022 23:21

Fix tidy

5384155

Fix compile

5662fdd

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 7, 2022

ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 7, 2022

Fix

96567cd

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Jun 7, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 7, 2022

ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 8, 2022

ti-chi-bot merged commit 5847f1c into pingcap:master Jun 8, 2022

solotzg deleted the pch branch June 8, 2022 01:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `Precompiled header` for modules & Refine by PImpl to accelerate release build #5047

Add `Precompiled header` for modules & Refine by PImpl to accelerate release build #5047

solotzg commented Jun 2, 2022 •

edited

Loading

ti-chi-bot commented Jun 2, 2022 •

edited

Loading

solotzg commented Jun 2, 2022

solotzg commented Jun 2, 2022

solotzg commented Jun 2, 2022

solotzg commented Jun 7, 2022

sre-bot commented Jun 7, 2022

solotzg commented Jun 7, 2022

ti-chi-bot commented Jun 7, 2022

ti-chi-bot commented Jun 7, 2022

solotzg commented Jun 7, 2022

sre-bot commented Jun 7, 2022

solotzg commented Jun 7, 2022

ti-chi-bot commented Jun 7, 2022

ti-chi-bot commented Jun 7, 2022

sre-bot commented Jun 7, 2022

solotzg commented Jun 8, 2022

Add Precompiled header for modules & Refine by PImpl to accelerate release build #5047

Add Precompiled header for modules & Refine by PImpl to accelerate release build #5047

Conversation

solotzg commented Jun 2, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Jun 2, 2022 • edited Loading

solotzg commented Jun 2, 2022

solotzg commented Jun 2, 2022

solotzg commented Jun 2, 2022

solotzg commented Jun 7, 2022

sre-bot commented Jun 7, 2022

Coverage for changed files

Coverage summary

solotzg commented Jun 7, 2022

ti-chi-bot commented Jun 7, 2022

ti-chi-bot commented Jun 7, 2022

solotzg commented Jun 7, 2022

sre-bot commented Jun 7, 2022

Coverage for changed files

Coverage summary

solotzg commented Jun 7, 2022

ti-chi-bot commented Jun 7, 2022

ti-chi-bot commented Jun 7, 2022

sre-bot commented Jun 7, 2022

Coverage for changed files

Coverage summary

solotzg commented Jun 8, 2022

Add `Precompiled header` for modules & Refine by PImpl to accelerate release build #5047

Add `Precompiled header` for modules & Refine by PImpl to accelerate release build #5047

solotzg commented Jun 2, 2022 •

edited

Loading

ti-chi-bot commented Jun 2, 2022 •

edited

Loading