Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhanced Features in 'Import Into' SQL Statement #46704

Closed
D3Hunter opened this issue Sep 6, 2023 · 1 comment
Closed

Enhanced Features in 'Import Into' SQL Statement #46704

D3Hunter opened this issue Sep 6, 2023 · 1 comment
Labels
type/feature-request Categorizes issue or PR as related to a new feature.

Comments

@D3Hunter
Copy link
Contributor

D3Hunter commented Sep 6, 2023

Feature Request

Is your feature request related to a problem? Please describe:

Describe the feature you'd like:

The 'import into' SQL was introduced in version 7.2, integrating the Lightning local backend (physical import mode). In version 7.2, it was released as an experimental feature and Only support 1 TiB source data and can’t resume from breakpoint. So we have to deliver the following new features to make "import into " more powerful.

This newly added feature in version 7.4 :

  1. Support "S3 global sort" to avoid problems caused by data range overlap, to make import more stable.
  2. support parameters like "SPLIT_FILE", and support import compressed source files like .gzip, .gz, .zstd, .zst and .snappy.
  3. Expected to support data imports of up to 100 TiB.(The final size depends on the results of testing.)
  4. The import performance can reach up to 1 TiB per hour.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

@D3Hunter D3Hunter added the type/feature-request Categorizes issue or PR as related to a new feature. label Sep 6, 2023
ti-chi-bot bot pushed a commit that referenced this issue Sep 11, 2023
ti-chi-bot bot pushed a commit that referenced this issue Sep 11, 2023
ti-chi-bot bot pushed a commit that referenced this issue Sep 20, 2023
ti-chi-bot bot pushed a commit that referenced this issue Oct 10, 2023
@D3Hunter
Copy link
Contributor Author

this part is release in v7.4.0, so close it

wuhuizuo pushed a commit to wuhuizuo/tidb that referenced this issue Apr 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/feature-request Categorizes issue or PR as related to a new feature.
Projects
None yet
Development

No branches or pull requests

1 participant