Fix blobstore truncate size may not right (#5127) #5147

ti-chi-bot · 2022-06-14T07:07:10Z

This is an automated cherry-pick of #5127

What problem does this PR solve?

Issue Number: close #5076 , close #5134

Problem Summary:

DB::Exception: No enough data in file /data0/db/page/log/blobfile_6, read bytes: 0 , expected bytes: 43
In getGcStats, BlobStore will call the getRightMargin to get the last free space and truncate the file.
But getRightMargin will return a invalid value if last free space is not align to end.
For example:
- Space limit is 100. So current space is [0, 100]
- mark a span {offset=90, size=10} as used, then the free range in SpaceMap is [0, 90).
- Then we should get 100 when we call getRightMargin
- But it return 0. Because in getRightMargin, we will get the last free space, and return it start.

What is changed and how it works?

Fix the getRightMargin
Change the log level when we do truncate.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

close pingcap#5076, close pingcap#5134

ti-chi-bot · 2022-06-14T07:07:12Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

JaySon-Huang
jiaqizho

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

JaySon-Huang

LGTM

JaySon-Huang · 2022-06-14T07:59:28Z

/run-all-tests

sre-bot · 2022-06-14T08:18:15Z

Coverage for changed files

Filename                        Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BlobStore.cpp                       567               164    71.08%          56                 2    96.43%        1188               270    77.27%         360               133    63.06%
PageDirectory.cpp                   571                83    85.46%          32                 1    96.88%        1153               177    84.65%         434                97    77.65%
spacemap/SpaceMap.h                   8                 3    62.50%           4                 1    75.00%          20                 7    65.00%           8                 2    75.00%
spacemap/SpaceMapBig.h               31                11    64.52%          11                 4    63.64%          70                43    38.57%          14                 7    50.00%
spacemap/SpaceMapRBTree.cpp         346                91    73.70%          18                 0   100.00%         572               108    81.12%         202                62    69.31%
spacemap/SpaceMapRBTree.h             2                 0   100.00%           2                 0   100.00%           5                 0   100.00%           0                 0         -
spacemap/SpaceMapSTDMap.h           161                18    88.82%          11                 0   100.00%         264                17    93.56%         102                18    82.35%
tests/gtest_free_map.cpp           2506               408    83.72%          14                 0   100.00%         298                 0   100.00%         814               407    50.00%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                              4192               778    81.44%         148                 8    94.59%        3570               622    82.58%        1934               726    62.46%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18242      9787             46.35%    204431  98029        52.05%

full coverage report (for internal network access only)

jiaqizho · 2022-06-15T02:46:53Z

/merge

ti-chi-bot · 2022-06-15T02:46:54Z

@jiaqizho: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-06-15T02:46:55Z

This pull request has been accepted and is ready to merge.

Commit hash: 5825bd0

sre-bot · 2022-06-15T02:55:47Z

Coverage for changed files

Filename                        Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BlobStore.cpp                       567               164    71.08%          56                 2    96.43%        1188               270    77.27%         360               133    63.06%
PageDirectory.cpp                   571                83    85.46%          32                 1    96.88%        1153               177    84.65%         434                97    77.65%
spacemap/SpaceMap.h                   8                 3    62.50%           4                 1    75.00%          20                 7    65.00%           8                 2    75.00%
spacemap/SpaceMapBig.h               31                11    64.52%          11                 4    63.64%          70                43    38.57%          14                 7    50.00%
spacemap/SpaceMapRBTree.cpp         346                91    73.70%          18                 0   100.00%         572               108    81.12%         202                62    69.31%
spacemap/SpaceMapRBTree.h             2                 0   100.00%           2                 0   100.00%           5                 0   100.00%           0                 0         -
spacemap/SpaceMapSTDMap.h           161                18    88.82%          11                 0   100.00%         264                17    93.56%         102                18    82.35%
tests/gtest_free_map.cpp           2506               408    83.72%          14                 0   100.00%         298                 0   100.00%         814               407    50.00%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                              4192               778    81.44%         148                 8    94.59%        3570               622    82.58%        1934               726    62.46%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18242      9787             46.35%    204431  98028        52.05%

full coverage report (for internal network access only)

jiaqizho · 2022-06-16T02:24:29Z

/merge

ti-chi-bot · 2022-06-16T02:24:30Z

@jiaqizho: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sre-bot · 2022-06-16T02:35:50Z

Coverage for changed files

Filename                        Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BlobStore.cpp                       567               164    71.08%          56                 2    96.43%        1188               270    77.27%         360               133    63.06%
PageDirectory.cpp                   571                83    85.46%          32                 1    96.88%        1153               177    84.65%         434                97    77.65%
spacemap/SpaceMap.h                   8                 3    62.50%           4                 1    75.00%          20                 7    65.00%           8                 2    75.00%
spacemap/SpaceMapBig.h               31                11    64.52%          11                 4    63.64%          70                43    38.57%          14                 7    50.00%
spacemap/SpaceMapRBTree.cpp         346                91    73.70%          18                 0   100.00%         572               108    81.12%         202                62    69.31%
spacemap/SpaceMapRBTree.h             2                 0   100.00%           2                 0   100.00%           5                 0   100.00%           0                 0         -
spacemap/SpaceMapSTDMap.h           161                18    88.82%          11                 0   100.00%         264                17    93.56%         102                18    82.35%
tests/gtest_free_map.cpp           2506               408    83.72%          14                 0   100.00%         298                 0   100.00%         814               407    50.00%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                              4192               778    81.44%         148                 8    94.59%        3570               622    82.58%        1934               726    62.46%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18242      9787             46.35%    204431  98042        52.04%

full coverage report (for internal network access only)

jiaqizho · 2022-06-17T03:09:18Z

/merge

ti-chi-bot · 2022-06-17T03:09:19Z

@jiaqizho: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sre-bot · 2022-06-17T03:18:49Z

Coverage for changed files

Filename                        Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BlobStore.cpp                       567               164    71.08%          56                 2    96.43%        1188               270    77.27%         360               133    63.06%
PageDirectory.cpp                   571                83    85.46%          32                 1    96.88%        1153               177    84.65%         434                97    77.65%
spacemap/SpaceMap.h                   8                 3    62.50%           4                 1    75.00%          20                 7    65.00%           8                 2    75.00%
spacemap/SpaceMapBig.h               31                11    64.52%          11                 4    63.64%          70                43    38.57%          14                 7    50.00%
spacemap/SpaceMapRBTree.cpp         346                91    73.70%          18                 0   100.00%         572               108    81.12%         202                62    69.31%
spacemap/SpaceMapRBTree.h             2                 0   100.00%           2                 0   100.00%           5                 0   100.00%           0                 0         -
spacemap/SpaceMapSTDMap.h           161                18    88.82%          11                 0   100.00%         264                17    93.56%         102                18    82.35%
tests/gtest_free_map.cpp           2506               408    83.72%          14                 0   100.00%         298                 0   100.00%         814               407    50.00%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                              4192               778    81.44%         148                 8    94.59%        3570               622    82.58%        1934               726    62.46%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18242      9787             46.35%    204431  98028        52.05%

full coverage report (for internal network access only)

jiaqizho · 2022-06-30T09:31:08Z

/merge

JaySon-Huang · 2022-08-17T11:27:50Z

/merge

ti-chi-bot · 2022-08-17T11:27:51Z

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

JaySon-Huang · 2022-08-17T13:30:06Z

/rebuild

JaySon-Huang · 2022-08-17T13:30:16Z

/run-all-tests

JaySon-Huang · 2022-08-18T03:04:59Z

/rebuild

JaySon-Huang · 2022-08-18T03:05:16Z

/run-all-tests

sre-bot · 2022-08-18T03:17:59Z

Coverage for changed files

Filename                        Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BlobStore.cpp                       568               163    71.30%          57                 2    96.49%        1201               270    77.52%         360               128    64.44%
PageDirectory.cpp                   580                87    85.00%          32                 1    96.88%        1161               181    84.41%         440               100    77.27%
spacemap/SpaceMap.h                   8                 3    62.50%           4                 1    75.00%          20                 7    65.00%           8                 2    75.00%
spacemap/SpaceMapBig.h               31                11    64.52%          11                 4    63.64%          70                43    38.57%          14                 7    50.00%
spacemap/SpaceMapRBTree.cpp         346                91    73.70%          18                 0   100.00%         572               108    81.12%         202                62    69.31%
spacemap/SpaceMapRBTree.h             2                 0   100.00%           2                 0   100.00%           5                 0   100.00%           0                 0         -
spacemap/SpaceMapSTDMap.h           161                18    88.82%          11                 0   100.00%         264                17    93.56%         102                18    82.35%
tests/gtest_free_map.cpp           2506               408    83.72%          14                 0   100.00%         298                 0   100.00%         814               407    50.00%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                              4202               781    81.41%         149                 8    94.63%        3591               626    82.57%        1940               724    62.68%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18256      9782             46.42%    204824  98013        52.15%

full coverage report (for internal network access only)

Fix blobstore truncate size may not right (pingcap#5127)

5825bd0

close pingcap#5076, close pingcap#5134

ti-chi-bot mentioned this pull request Jun 14, 2022

Fix blobstore truncate size may not right #5127

Merged

12 tasks

ti-chi-bot assigned jiaqizho Jun 14, 2022

JaySon-Huang approved these changes Jun 14, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 14, 2022

jiaqizho approved these changes Jun 15, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 15, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 15, 2022

hehechen approved these changes Jun 17, 2022

View reviewed changes

VelocityLight added cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/cherry-pick-not-approved labels Aug 17, 2022

Merge branch 'release-6.1' into cherry-pick-5127-to-release-6.1

09b61b5

JaySon-Huang mentioned this pull request Aug 17, 2022

PageStorage: Fix entry.tag after full gc && add more debug message (#5094) #5096

Merged

12 tasks

Merge branch 'release-6.1' into cherry-pick-5127-to-release-6.1

f115a7f

ti-chi-bot merged commit e7deb68 into pingcap:release-6.1 Aug 18, 2022

jayl1e added this to the v6.1.1 milestone Aug 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix blobstore truncate size may not right (#5127) #5147

Fix blobstore truncate size may not right (#5127) #5147

ti-chi-bot commented Jun 14, 2022

ti-chi-bot commented Jun 14, 2022 •

edited

Loading

JaySon-Huang left a comment

JaySon-Huang commented Jun 14, 2022

sre-bot commented Jun 14, 2022

jiaqizho commented Jun 15, 2022

ti-chi-bot commented Jun 15, 2022

ti-chi-bot commented Jun 15, 2022

sre-bot commented Jun 15, 2022

jiaqizho commented Jun 16, 2022

ti-chi-bot commented Jun 16, 2022

sre-bot commented Jun 16, 2022

jiaqizho commented Jun 17, 2022

ti-chi-bot commented Jun 17, 2022

sre-bot commented Jun 17, 2022

jiaqizho commented Jun 30, 2022

JaySon-Huang commented Aug 17, 2022

ti-chi-bot commented Aug 17, 2022

JaySon-Huang commented Aug 17, 2022

JaySon-Huang commented Aug 17, 2022

JaySon-Huang commented Aug 18, 2022

JaySon-Huang commented Aug 18, 2022

sre-bot commented Aug 18, 2022

Fix blobstore truncate size may not right (#5127) #5147

Fix blobstore truncate size may not right (#5127) #5147

Conversation

ti-chi-bot commented Jun 14, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Jun 14, 2022 • edited Loading

JaySon-Huang left a comment

Choose a reason for hiding this comment

JaySon-Huang commented Jun 14, 2022

sre-bot commented Jun 14, 2022

Coverage for changed files

Coverage summary

jiaqizho commented Jun 15, 2022

ti-chi-bot commented Jun 15, 2022

ti-chi-bot commented Jun 15, 2022

sre-bot commented Jun 15, 2022

Coverage for changed files

Coverage summary

jiaqizho commented Jun 16, 2022

ti-chi-bot commented Jun 16, 2022

sre-bot commented Jun 16, 2022

Coverage for changed files

Coverage summary

jiaqizho commented Jun 17, 2022

ti-chi-bot commented Jun 17, 2022

sre-bot commented Jun 17, 2022

Coverage for changed files

Coverage summary

jiaqizho commented Jun 30, 2022

JaySon-Huang commented Aug 17, 2022

ti-chi-bot commented Aug 17, 2022

JaySon-Huang commented Aug 17, 2022

JaySon-Huang commented Aug 17, 2022

JaySon-Huang commented Aug 18, 2022

JaySon-Huang commented Aug 18, 2022

sre-bot commented Aug 18, 2022

Coverage for changed files

Coverage summary

ti-chi-bot commented Jun 14, 2022 •

edited

Loading