Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add dumpling into tidb tools #3453

Merged
merged 23 commits into from
Jul 31, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions TOC.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,12 @@
+ [Use TiDB Ansible](/scale-tidb-using-ansible.md)
+ [Use TiDB Operator](https://docs.pingcap.com/tidb-in-kubernetes/v1.1/scale-a-tidb-cluster)
+ Backup and Restore
+ [Use Mydumper and TiDB Lightning](/backup-and-restore-using-mydumper-lightning.md)
+ [Use Dumpling for Export or Backup](/export-or-backup-using-dumpling.md)
+ Use BR Tool
+ Use BR Tool (Recommended)
+ [Use BR Tool](/br/backup-and-restore-tool.md)
+ [BR Use Cases](/br/backup-and-restore-use-cases.md)
+ [BR storages](/br/backup-and-restore-storages.md)
+ [Use Dumpling and TiDB Lightning (Recommended)](/backup-and-restore-using-dumpling-lightning.md)
+ [Use Mydumper and TiDB Lightning](/backup-and-restore-using-mydumper-lightning.md)
+ [Read Historical Data](/read-historical-data.md)
+ [Configure Time Zone](/configure-time-zone.md)
+ [Daily Checklist](/daily-check.md)
Expand Down Expand Up @@ -186,6 +186,7 @@
+ [FAQ](/tidb-lightning/tidb-lightning-faq.md)
+ [Glossary](/tidb-lightning/tidb-lightning-glossary.md)
+ [TiCDC](/ticdc/ticdc-overview.md)
+ [Dumpling](/dumpling-overview.md)
+ sync-diff-inspector
+ [Overview](/sync-diff-inspector/sync-diff-inspector-overview.md)
+ [Data Check for Tables with Different Schema/Table Names](/sync-diff-inspector/route-diff.md)
Expand Down
86 changes: 86 additions & 0 deletions backup-and-restore-using-dumpling-lightning.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
---
title: Use Dumpling and TiDB Lightning for Data Backup and Restoration
summary: Introduce how to use Dumpling and TiDB Lightning to backup and restore full data of TiDB.
aliases: ['/docs-cn/dev/export-or-backup-using-dumpling/','/zh/tidb/dev/export-or-backup-using-dumpling']
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed the two aliases in a0ec32a.

---

# Use Dumpling and TiDB Lightning for Data Backup and Restoration

This document introduces in detail how to use Dumpling and TiDB Lightning to backup and restore full data of TiDB. For incremental backup and replication to downstream, refer to [TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md).

Suppose that the TiDB server information is as follows:

|Server Name|Server Address|Port|User|Password|
|----|-------|----|----|--------|
|TiDB|127.0.0.1|4000|root|*|

Use the following tools for data backup and restoration:

- [Dumpling](/dumpling-overview.md): to export data from TiDB
- [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md): to import data into TiDB

## Best practices for full backup and restoration using Dumpling/TiDB Lightning

To quickly backup and restore data (especially large amounts of data), refer to the following recommendations:

* Keep the exported data file as small as possible. It is recommended to use the `-F` option of Dumpling to set the file size. If you use TiDB Lightning to restore data, it is recommended that you set the value of `-F` to `256m`.
* If some of the exported tables have many rows, you can enable concurrency in the table by setting the `-r` option.

## Backup data from TiDB

Use the following `dumpling` command to backup data from TiDB.

{{< copyable "shell-regular" >}}

```bash
./bin/dumpling -h 127.0.0.1 -P 4000 -u root -t 32 -F 256m -T test.t1 -T test.t2 -o ./var/test
```

In this command:

- `-T test.t1 -T test.t2` means that only the two tables `test`.`t1` and `test`.`t2` are exported. For more methods to filter exported data, refer to [Filter exported data](/dumpling-overview.md#filter-the-exported-data).
- `-t 32` means that 32 threads are used to export the data.
- `-F 256m` means that a table is partitioned into chunks, and one chunk is 256MB.

Starting from v4.0.0, Dumpling can automatically extends the GC time if it can access the PD address of the TiDB cluster. But for TiDB earlier than v4.0.0, you need to manually modify the GC time. Otherwise, you might bump into the following error:

```log
Could not read data from testSchema.testTable: GC life time is shorter than transaction duration, transaction starts at 2019-08-05 21:10:01.451 +0800 CST, GC safe point is 2019-08-05 21:14:53.801 +0800 CST
```

The steps to manually modify the GC time are as follows:

1. Before executing the `dumpling` command, query the [GC](/garbage-collection-overview.md) value of the TiDB cluster and execute the following statement in the MySQL client to adjust it to a suitable value:

{{< copyable "sql" >}}

```sql
SELECT * FROM mysql.tidb WHERE VARIABLE_NAME = 'tikv_gc_life_time';
```

```sql
+-----------------------+------------------------------------------------------------------------------------------------+
| VARIABLE_NAME | VARIABLE_VALUE |
+-----------------------+------------------------------------------------------------------------------------------------+
| tikv_gc_life_time | 10m0s |
+-----------------------+------------------------------------------------------------------------------------------------+
1 rows in set (0.02 sec)
```

{{< copyable "sql" >}}

```sql
update mysql.tidb set VARIABLE_VALUE = '720h' where VARIABLE_NAME = 'tikv_gc_life_time';
```

2. After executing the `dumpling` command, restore the GC value of the TiDB cluster to the initial value in step 1:

{{< copyable "sql" >}}

```sql
update mysql.tidb set VARIABLE_VALUE = '10m' where VARIABLE_NAME = 'tikv_gc_life_time';
```

## Restore data into TiDB

To restore data into TiDB, use TiDB Lightning to import the exported data. See [TiDB Lightning Tutorial](/tidb-lightning/tidb-lightning-tidb-backend.md).
20 changes: 10 additions & 10 deletions backup-and-restore-using-mydumper-lightning.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ aliases: ['/docs/dev/backup-and-restore-using-mydumper-lightning/','/docs/dev/ho

This document describes how to perform full backup and restoration of the TiDB data using Mydumper and TiDB Lightning. For incremental backup and restoration, refer to [TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md).

Suppose that the TiDB service information is as follows:
Suppose that the TiDB server information is as follows:

|Name|Address|Port|User|Password|
|Server Name|Server Address|Port|User|Password|
|:----|:-------|:----|:----|:--------|
|TiDB|127.0.0.1|4000|root|*|

Expand All @@ -32,25 +32,25 @@ Use [Mydumper](/mydumper-overview.md) to export data from TiDB and use [TiDB Lig

To quickly backup and restore data (especially large amounts of data), refer to the following recommendations:

* Keep the exported data file as small as possible. It is recommended to use the `-F` parameter to set the file size. If you use TiDB Lightning to restore data, it is recommended that you set the value of `-F` to `256` (MB). If you use `loader` for restoration, it is recommended to set the value to `64` (MB).
* Keep the exported data file as small as possible. It is recommended to use the `-F` option of Mydumper to set the file size. If you use TiDB Lightning to restore data, it is recommended that you set the value of `-F` to `256` (MB). If you use `loader` for restoration, it is recommended to set the value to `64` (MB).

## Backup data from TiDB

Use `mydumper` to backup data from TiDB.
Use the following `mydumper` command to backup data from TiDB:

{{< copyable "shell-regular" >}}

```bash
./bin/mydumper -h 127.0.0.1 -P 4000 -u root -t 32 -F 256 -B test -T t1,t2 --skip-tz-utc -o ./var/test
```

In this command,
In this command:

`-B test` means that the data is exported from the `test` database.
`-T t1,t2` means that only the `t1` and `t2` tables are exported.
`-t 32` means that 32 threads are used to export the data.
`-F 256` means that a table is partitioned into chunks, and one chunk is 256MB.
`--skip-tz-utc` means to ignore the inconsistency of time zone setting between MySQL and the data exporting machine and to disable automatic conversion.
- `-B test` means that the data is exported from the `test` database.
- `-T t1,t2` means that only the `t1` and `t2` tables are exported.
- `-t 32` means that 32 threads are used to export the data.
- `-F 256` means that a table is partitioned into chunks, and one chunk is 256MB.
- `--skip-tz-utc` means to ignore the inconsistency of time zone setting between MySQL and the data exporting machine and to disable automatic conversion.

If `mydumper` returns the following error:

Expand Down
2 changes: 1 addition & 1 deletion br/backup-and-restore-tool.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ aliases: ['/docs/dev/br/backup-and-restore-tool/','/docs/dev/reference/tools/br/

# Use BR to Back up and Restore Data

[Backup & Restore](http://github.com/pingcap/br) (BR) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with [`dumpling`](/export-or-backup-using-dumpling.md) and [`mydumper`/`loader`](/backup-and-restore-using-mydumper-lightning.md), BR is more suitable for scenarios of huge data volume. This document describes the BR command line, detailed use examples, best practices, restrictions, and introduces the implementation principles of BR.
[Backup & Restore](http://github.com/pingcap/br) (BR) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with [`dumpling`](/backup-and-restore-using-dumpling-lightning.md) and [`mydumper`/`loader`](/backup-and-restore-using-mydumper-lightning.md), BR is more suitable for scenarios of huge data volume. This document describes the BR command line, detailed use examples, best practices, restrictions, and introduces the implementation principles of BR.

## Usage restrictions

Expand Down
14 changes: 14 additions & 0 deletions download-ecosystem-tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,20 @@ Download [DM](https://docs.pingcap.com/tidb-data-migration/v1.0/overview) by usi
>
> `{version}` in the above download link indicates the version number of DM. For example, the download link for `v1.0.1` is `https://download.pingcap.org/dm-v1.0.1-linux-amd64.tar.gz`. You can check the published DM versions in the [DM Release](https://github.com/pingcap/dm/releases) page.

## Dumpling

Download [Dumpling](/dumpling-overview.md) from the links below:

| Installation package | Operating system | Architecture | SHA256 checksum |
|:---|:---|:---|:---|
| `https://download.pingcap.org/tidb-toolkit-{version}-linux-amd64.tar.gz` | Linux | amd64 | `https://download.pingcap.org/tidb-toolkit-{version}-linux-amd64.sha256` |

> **Note:**
>
> The `{version}` in the download link is the version number of Dumpling. For example, the link for downloading the `v4.0.2` version of Dumpling is `https://download.pingcap.org/tidb-toolkit-v4.0.2-linux-amd64.tar.gz`. You can view the currently released versions in [Dumpling Releases](https://github.com/pingcap/dumpling/releases).
>
> Dumpling supports arm64 linux. You can replace `amd64` in the download link with `arm64`, which means the `arm64` version of Dumpling.
lilin90 marked this conversation as resolved.
Show resolved Hide resolved

## Syncer, Loader, and Mydumper

If you want to download the latest version of [Syncer](/syncer-overview.md), [Loader](/loader-overview.md), or [Mydumper](/mydumper-overview.md), directly download the tidb-enterprise-tools package, because all these tools are included in this package.
Expand Down
Loading