Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

br/restore: use a new algorithm for batching, deprecating --ddl-batch-size #55809

Closed
wants to merge 3 commits into from

Conversation

YuJuncen
Copy link
Contributor

@YuJuncen YuJuncen commented Sep 3, 2024

What problem does this PR solve?

Issue Number: close #55639

Problem Summary:
See the issue.

What changed and how does it work?

Now, we will fetch the max raft entry size from TiKV. And try to batching to the size to 7/8 of the max raft entry size.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)

I have tested restoring about 500k tables. The total number of DDLs decreased about 1/2. But the speed of creating table not increased. (About 32 mins for creating all tables for both master and this patch.).

  • No need to test
    • I checked and no code files have been changed.

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

By using a new algorithm for batching `CREATE TABLE` DDLs, we have reduced the number of jobs needed for creating tables. Also the old configuration `--ddl-batch-size` was deprecated.

Signed-off-by: hillium <yujuncen@pingcap.com>
Signed-off-by: hillium <yujuncen@pingcap.com>
Signed-off-by: hillium <yujuncen@pingcap.com>
@ti-chi-bot ti-chi-bot bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Sep 3, 2024
Copy link

ti-chi-bot bot commented Sep 3, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Copy link

ti-chi-bot bot commented Sep 3, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from yujuncen, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 3, 2024
Copy link

tiprow bot commented Sep 3, 2024

Hi @YuJuncen. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 28, 2024
Copy link

ti-chi-bot bot commented Sep 28, 2024

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@YuJuncen YuJuncen closed this Oct 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

br/restore: remove the --ddl-batch-size flag, or make it size-based instead of length-based
1 participant