-
Notifications
You must be signed in to change notification settings - Fork 678
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add dumpling into tidb tools #3453
Merged
Merged
Changes from 19 commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
e46b07f
align the structure of https://github.com/pingcap/docs-cn/pull/3800/
98a0add
revert
d5c8f1d
Merge remote-tracking branch 'upstream/master' into update-dumpling
943add0
Merge remote-tracking branch 'upstream/master' into update-dumpling
0a85f78
Merge branch 'master' into update-dumpling
yikeke a42e50f
Update dumpling-overview.md
fe6f352
Update dumpling-overview.md
5ca54ef
update backup-and-restore-using-dumpling-lightning.md
8ba56ea
ADD summary
a53014e
Merge branch 'master' into update-dumpling
yikeke e3d1fb9
fix an anchor
7b3fddb
Update dumpling-overview.md
yikeke fbaadca
Apply suggestions from code review
yikeke 95cc55d
Apply suggestions from code review
yikeke b05cfd3
Update dumpling-overview.md
yikeke aa671d6
Apply suggestions from code review
yikeke aa7b9b0
Apply suggestions from code review
yikeke 8143735
Update dumpling-overview.md
yikeke 303f710
parameter -> option
dd37a73
Update note format and add inline code format
lilin90 f65b608
Merge branch 'master' into update-dumpling
lilin90 68fc911
Merge branch 'master' into update-dumpling
lilin90 f1ca496
fix 2 dead links from upstream
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,86 @@ | ||
--- | ||
title: Use Dumpling and TiDB Lightning for Data Backup and Restoration | ||
summary: Introduce how to use Dumpling and TiDB Lightning to backup and restore full data of TiDB. | ||
aliases: ['/docs-cn/dev/export-or-backup-using-dumpling/','/zh/tidb/dev/export-or-backup-using-dumpling'] | ||
--- | ||
|
||
# Use Dumpling and TiDB Lightning for Data Backup and Restoration | ||
|
||
This document introduces in detail how to use Dumpling and TiDB Lightning to backup and restore full data of TiDB. For incremental backup and replication to downstream, refer to [TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md). | ||
|
||
Suppose that the TiDB server information is as follows: | ||
|
||
|Server Name|Server Address|Port|User|Password| | ||
|----|-------|----|----|--------| | ||
|TiDB|127.0.0.1|4000|root|*| | ||
|
||
Use the following tools for data backup and restoration: | ||
|
||
- [Dumpling](/dumpling-overview.md): to export data from TiDB | ||
- [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md): to import data into TiDB | ||
|
||
## Best practices for full backup and restoration using Dumpling/TiDB Lightning | ||
|
||
To quickly backup and restore data (especially large amounts of data), refer to the following recommendations: | ||
|
||
* Keep the exported data file as small as possible. It is recommended to use the `-F` option of Dumpling to set the file size. If you use TiDB Lightning to restore data, it is recommended that you set the value of `-F` to `256m`. | ||
* If some of the exported tables have many rows, you can enable concurrency in the table by setting the `-r` option. | ||
|
||
## Backup data from TiDB | ||
|
||
Use the following `dumpling` command to backup data from TiDB. | ||
|
||
{{< copyable "shell-regular" >}} | ||
|
||
```bash | ||
./bin/dumpling -h 127.0.0.1 -P 4000 -u root -t 32 -F 256m -T test.t1 -T test.t2 -o ./var/test | ||
``` | ||
|
||
In this command: | ||
|
||
- `-T test.t1 -T test.t2` means that only the two tables `test`.`t1` and `test`.`t2` are exported. For more methods to filter exported data, refer to [Filter exported data](/dumpling-overview.md#filter-the-exported-data). | ||
- `-t 32` means that 32 threads are used to export the data. | ||
- `-F 256m` means that a table is partitioned into chunks, and one chunk is 256MB. | ||
|
||
Starting from v4.0.0, Dumpling can automatically extends the GC time if it can access the PD address of the TiDB cluster. But for TiDB earlier than v4.0.0, you need to manually modify the GC time. Otherwise, you might bump into the following error: | ||
|
||
```log | ||
Could not read data from testSchema.testTable: GC life time is shorter than transaction duration, transaction starts at 2019-08-05 21:10:01.451 +0800 CST, GC safe point is 2019-08-05 21:14:53.801 +0800 CST | ||
``` | ||
|
||
The steps to manually modify the GC time are as follows: | ||
|
||
1. Before executing the `dumpling` command, query the [GC](/garbage-collection-overview.md) value of the TiDB cluster and execute the following statement in the MySQL client to adjust it to a suitable value: | ||
|
||
{{< copyable "sql" >}} | ||
|
||
```sql | ||
SELECT * FROM mysql.tidb WHERE VARIABLE_NAME = 'tikv_gc_life_time'; | ||
``` | ||
|
||
```sql | ||
+-----------------------+------------------------------------------------------------------------------------------------+ | ||
| VARIABLE_NAME | VARIABLE_VALUE | | ||
+-----------------------+------------------------------------------------------------------------------------------------+ | ||
| tikv_gc_life_time | 10m0s | | ||
+-----------------------+------------------------------------------------------------------------------------------------+ | ||
1 rows in set (0.02 sec) | ||
``` | ||
|
||
{{< copyable "sql" >}} | ||
|
||
```sql | ||
update mysql.tidb set VARIABLE_VALUE = '720h' where VARIABLE_NAME = 'tikv_gc_life_time'; | ||
``` | ||
|
||
2. After executing the `dumpling` command, restore the GC value of the TiDB cluster to the initial value in step 1: | ||
|
||
{{< copyable "sql" >}} | ||
|
||
```sql | ||
update mysql.tidb set VARIABLE_VALUE = '10m' where VARIABLE_NAME = 'tikv_gc_life_time'; | ||
``` | ||
|
||
## Restore data into TiDB | ||
|
||
To restore data into TiDB, use TiDB Lightning to import the exported data. See [TiDB Lightning Tutorial](/tidb-lightning/tidb-lightning-tidb-backend.md). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed the two aliases in a0ec32a.