Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: batch processing overview and pipelined dml #19818

Merged
merged 17 commits into from
Jan 8, 2025

Conversation

qiancai
Copy link
Collaborator

@qiancai qiancai commented Dec 26, 2024

First-time contributors' checklist

What is changed, added or deleted? (Required)

  1. A batch processing overview doc to disambiguate features
  2. A standalone doc for Pipelined DML

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions (in Chinese).

  • master (the latest development version)
  • v8.5 (TiDB 8.5 versions)
  • v8.4 (TiDB 8.4 versions)
  • v8.3 (TiDB 8.3 versions)
  • v8.2 (TiDB 8.2 versions)
  • v8.1 (TiDB 8.1 versions)
  • v7.5 (TiDB 7.5 versions)
  • v7.1 (TiDB 7.1 versions)
  • v6.5 (TiDB 6.5 versions)
  • v6.1 (TiDB 6.1 versions)
  • v5.4 (TiDB 5.4 versions)
  • v5.3 (TiDB 5.3 versions)

What is the related PR or file link(s)?

Do your changes match any of the following descriptions?

  • Delete files
  • Change aliases
  • Need modification after applied to another branch
  • Might cause conflicts after applied to another branch

@qiancai qiancai added area/transaction Indicates that the Issue or PR belongs to the area of transaction. needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. translation/from-docs-cn This PR is translated from a PR in pingcap/docs-cn. type/enhancement The issue or PR belongs to an enhancement. v8.5 labels Dec 26, 2024
@ti-chi-bot ti-chi-bot bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Dec 26, 2024
@qiancai qiancai marked this pull request as draft December 26, 2024 10:19
@ti-chi-bot ti-chi-bot bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 26, 2024
@ti-chi-bot ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Dec 27, 2024
pipelined-dml.md.md Outdated Show resolved Hide resolved
@qiancai qiancai requested a review from ekexium December 27, 2024 08:56
@qiancai qiancai self-assigned this Dec 27, 2024
pipelined-dml.md.md Outdated Show resolved Hide resolved
github-actions bot pushed a commit to qiancai/pingcap-docsite-preview that referenced this pull request Dec 27, 2024
github-actions bot pushed a commit to qiancai/pingcap-docsite-preview that referenced this pull request Dec 27, 2024
TOC.md Outdated
@@ -397,6 +397,7 @@
- [Use Load Base Split](/configure-load-base-split.md)
- [Use Store Limit](/configure-store-limit.md)
- [DDL Execution Principles and Best Practices](/ddl-introduction.md)
- [Batch Data Processing](/batch-processing.md)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Batch processing is a more commonly used term, I suppose?

batch-processing.md Show resolved Hide resolved
batch-processing.md Outdated Show resolved Hide resolved
pipelined-dml.md.md Outdated Show resolved Hide resolved

- Memory limits: traditional DML operations might encounter out-of-memory (OOM) errors when handling large datasets.
- Performance bottlenecks: large transactions are often inefficient and is prone to causing workload fluctuations.
- Operational limits: TiDB memory limits make it difficult to execute ultra-large data processing tasks.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about removing this point as it's a duplicate of the 1st point? Also for the Chinese doc.

pipelined-dml.md.md Outdated Show resolved Hide resolved
Co-authored-by: ekexium <eke@fastmail.com>
github-actions bot pushed a commit to qiancai/pingcap-docsite-preview that referenced this pull request Jan 6, 2025
TOC.md Outdated Show resolved Hide resolved
batch-processing.md Outdated Show resolved Hide resolved
batch-processing.md Outdated Show resolved Hide resolved
batch-processing.md Outdated Show resolved Hide resolved
batch-processing.md Outdated Show resolved Hide resolved
pipelined-dml.md.md Outdated Show resolved Hide resolved
pipelined-dml.md.md Outdated Show resolved Hide resolved
batch-processing.md Outdated Show resolved Hide resolved
batch-processing.md Outdated Show resolved Hide resolved
pipelined-dml.md.md Outdated Show resolved Hide resolved
pipelined-dml.md.md Outdated Show resolved Hide resolved
pipelined-dml.md.md Outdated Show resolved Hide resolved
pipelined-dml.md.md Outdated Show resolved Hide resolved
pipelined-dml.md.md Outdated Show resolved Hide resolved
@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jan 8, 2025
Copy link

ti-chi-bot bot commented Jan 8, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-01-08 02:29:55.104072244 +0000 UTC m=+320738.392903950: ☑️ agreed by Oreoxmt.

@ti-chi-bot ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 8, 2025
batch-processing.md Outdated Show resolved Hide resolved
Copy link

ti-chi-bot bot commented Jan 8, 2025

@ekexium: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Co-authored-by: ekexium <eke@fastmail.com>
@qiancai
Copy link
Collaborator Author

qiancai commented Jan 8, 2025

/approve

@qiancai qiancai added the lgtm label Jan 8, 2025
Copy link

ti-chi-bot bot commented Jan 8, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qiancai

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the approved label Jan 8, 2025
@ti-chi-bot ti-chi-bot bot merged commit 0d5fe90 into pingcap:master Jan 8, 2025
9 checks passed
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-8.5: #19938.

ti-chi-bot pushed a commit to ti-chi-bot/docs that referenced this pull request Jan 8, 2025
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
github-actions bot pushed a commit to qiancai/pingcap-docsite-preview that referenced this pull request Jan 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved area/transaction Indicates that the Issue or PR belongs to the area of transaction. lgtm needs-1-more-lgtm Indicates a PR needs 1 more LGTM. needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. translation/from-docs-cn This PR is translated from a PR in pingcap/docs-cn. type/enhancement The issue or PR belongs to an enhancement. v8.5
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants