Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support keyspace with S3 disagg read #7115

Merged
merged 12 commits into from
Mar 21, 2023
Merged

support keyspace with S3 disagg read #7115

merged 12 commits into from
Mar 21, 2023

Conversation

JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Mar 21, 2023 •
edited
Loading

What problem does this PR solve?

Issue Number: ref #6827

Problem Summary: We need to support keyspace for S3 disagg read. A following PR of #7113

What is changed and how it works?

Also fix the bug that in disagg mode, data in memtable is not read out.

Depends on:

Requests:

  • RN: Disagg read task handled in StorageDisaggregatedRemote, setup the keyspace_id and api_version
  • WN: Receive request by WNEstablishDisaggTaskHandler, parse the keyspace_id from meta and set it to DAGContext:: keyspace_id
  • WN: Run into DAGStorageInterpreter and get the DeltaMergeStorage by keyspace_id, table_id (reuse existing logic)
  • WN: DeltaMergeStore::writeNodeBuildRemoteReadSnapshot setup the keyspace_id, table_id along with the snapshot
  • WN: WNEstablishDisaggTaskHandler::execute serialize RemotePb::RemotePhysicalTable
  • RN: StorageDisaggregated::buildDisaggregatedTask parse the keyspace_id, table_id from RemotePb::RemotePhysicalTable and calls RNRemoteTableReadTask::buildFrom
  • RN: The keyspace_id is added into each RNRemoteSegmentReadTask

Local cache:

  • RN: fetch the page data from WN
  • RN: persist the page with ID{store_id, keyspace_id, table_id, page_id} in local cache

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Mar 21, 2023 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • flowbehappy
  • lidezhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 21, 2023
@JaySon-Huang
Copy link
Contributor Author

/run-all-tests

@JaySon-Huang JaySon-Huang changed the title [WIP] support keyspace with S3 disagg read support keyspace with S3 disagg read Mar 21, 2023
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 21, 2023
lidezhu
lidezhu approved these changes Mar 21, 2023
View reviewed changes
Copy link
Contributor

@lidezhu lidezhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 21, 2023
@flowbehappy flowbehappy self-requested a review March 21, 2023 09:15
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 21, 2023
@ti-chi-bot
Copy link
Member

@yongman: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@JaySon-Huang
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: f56548f

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 21, 2023
@JaySon-Huang
Copy link
Contributor Author

/hold
wait for the kvproto PR merge

@ti-chi-bot ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 21, 2023
@JaySon-Huang JaySon-Huang mentioned this pull request Mar 21, 2023
Merged
@ti-chi-bot ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Mar 21, 2023
@JaySon-Huang
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 21, 2023
@JaySon-Huang
Copy link
Contributor Author

/run-integration-test

@JaySon-Huang
Temporary disable async-grpc test
eedb2b8 
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@ti-chi-bot ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Mar 21, 2023
@JaySon-Huang
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: eedb2b8

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 21, 2023
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Mar 21, 2023 •
edited
Loading

temporary disable async-grpc test #7120

@ti-chi-bot ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Mar 21, 2023
@JaySon-Huang
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 0958cb0

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 21, 2023
@JaySon-Huang JaySon-Huang mentioned this pull request Mar 21, 2023
Closed
12 tasks
@ti-chi-bot
Copy link
Member

@JaySon-Huang: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

trigger some heavy tests which will not run always when PR updated.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@breezewish @JaySon-Huang
Fix memtable not read in disagg mode
9a35920 
Signed-off-by: Wish <breezewish@outlook.com>
@ti-chi-bot ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Mar 21, 2023
@JaySon-Huang JaySon-Huang mentioned this pull request Mar 21, 2023
Closed
12 tasks
@JaySon-Huang
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 9a35920

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 21, 2023
@lidezhu lidezhu mentioned this pull request Mar 21, 2023
Merged
12 tasks
@JaySon-Huang
Copy link
Contributor Author

/run-unit-test

@ti-chi-bot ti-chi-bot merged commit ddadc0c into pingcap:master Mar 21, 2023
@JaySon-Huang JaySon-Huang deleted the keyspace_s3 branch March 21, 2023 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@flowbehappy flowbehappy flowbehappy approved these changes

@yongman yongman yongman approved these changes

@lidezhu lidezhu lidezhu approved these changes

@breezewish breezewish Awaiting requested review from breezewish

@iosmanthus iosmanthus Awaiting requested review from iosmanthus

Assignees
No one assigned
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@JaySon-Huang @ti-chi-bot @flowbehappy @yongman @lidezhu @solotzg @breezewish