Skip to content
This repository has been archived by the owner on Jul 24, 2024. It is now read-only.

Create integration test for S3 storage #174

Merged
merged 8 commits into from
Mar 16, 2020
Merged

Conversation

kennytm
Copy link
Collaborator

@kennytm kennytm commented Feb 29, 2020

What problem does this PR solve?

Fix #157.

What is changed and how it works?

Copy the br_full test case but use S3 as storage.

We use minio as the local server. Added a script to automatically download minio as well as the TiDB components.

Check List

Tests

  • Integration test

Code changes

Side effects

Related changes

@kennytm

This comment has been minimized.

@kennytm
Copy link
Collaborator Author

kennytm commented Mar 2, 2020

incremental restoring an empty archive?
[2020-03-02T21:14:52.844Z] incremental restore start...
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.796 +08:00] [INFO] [version.go:26] ["Welcome to Backup & Restore (BR)"]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.796 +08:00] [INFO] [version.go:27] [BR] [release-version=None]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.796 +08:00] [INFO] [version.go:28] [BR] [git-hash=None]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.796 +08:00] [INFO] [version.go:29] [BR] [git-branch=None]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.796 +08:00] [INFO] [version.go:30] [BR] [go-version=go1.13]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.796 +08:00] [INFO] [version.go:31] [BR] [utc-build-time=None]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.796 +08:00] [INFO] [version.go:32] [BR] [race-enabled=false]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.796 +08:00] [INFO] [version.go:53] [arguments] [ca=] [cert=] [checksum=true] [concurrency=4] [db=br_incremental_index] [gcs.credentials-file=] [gcs.endpoint=] [gcs.predefined-acl=] [gcs.storage-class=] [help=false] [key=] [log-file=] [log-level=debug] [online=false] [pd="[127.0.0.1:2379]"] [ratelimit=0] [ratelimit-unit=1048576] [s3.acl=] [s3.endpoint=] [s3.provider=] [s3.region=] [s3.sse=] [s3.storage-class=] [send-credentials-to-tikv=true] [slow-log-file=] [status-addr=] [storage=local:///tmp/backup_restore_test/br_incremental_index/inc] [table=usertable]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.796 +08:00] [INFO] [client.go:134] ["[pd] create pd client with endpoints"] [pd-address="[127.0.0.1:2379]"]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.798 +08:00] [INFO] [base_client.go:242] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.798 +08:00] [INFO] [base_client.go:92] ["[pd] init cluster id"] [cluster-id=6799705913937919316]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.800 +08:00] [INFO] [client.go:134] ["[pd] create pd client with endpoints"] [pd-address="[127.0.0.1:2379]"]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.802 +08:00] [INFO] [base_client.go:242] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.802 +08:00] [INFO] [base_client.go:92] ["[pd] init cluster id"] [cluster-id=6799705913937919316]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.802 +08:00] [INFO] [conn.go:133] ["new mgr"] [pdAddrs=127.0.0.1:2379]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.806 +08:00] [INFO] [tidb.go:71] ["new domain"] [store=tikv-6799705913937919316] ["ddl lease"=1s] ["stats lease"=-1ns]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.820 +08:00] [INFO] [ddl.go:447] ["[ddl] start DDL"] [ID=005bb9b8-cb65-4aca-b8ee-85b6afdc2ddf] [runWorker=true]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.822 +08:00] [INFO] [ddl.go:435] ["[ddl] start delRangeManager OK"] ["is a emulator"=false]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.823 +08:00] [INFO] [ddl_worker.go:130] ["[ddl] start DDL worker"] [worker="worker 1, tp general"]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.823 +08:00] [INFO] [ddl_worker.go:130] ["[ddl] start DDL worker"] [worker="worker 2, tp add index"]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.823 +08:00] [DEBUG] [ddl.go:335] ["[ddl] check whether is the DDL owner"] [isOwner=false] [selfID=005bb9b8-cb65-4aca-b8ee-85b6afdc2ddf]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.823 +08:00] [DEBUG] [ddl.go:335] ["[ddl] check whether is the DDL owner"] [isOwner=false] [selfID=005bb9b8-cb65-4aca-b8ee-85b6afdc2ddf]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.830 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d4442730000000000fa0000000000000068] [nextEndKey=6d4442730000000000fa0000000000000069] [reverse=false] [txnStartTS=415022105965101060]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.833 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d44423a3236300000fd0000000000000068] [nextEndKey=6d44423a3236300000fd0000000000000069] [reverse=false] [txnStartTS=415022105965101060]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.833 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d44423a3100000000fb0000000000000068] [nextEndKey=6d44423a3100000000fb0000000000000069] [reverse=false] [txnStartTS=415022105965101060]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.833 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d44423a3300000000fb0000000000000068] [nextEndKey=6d44423a3300000000fb0000000000000069] [reverse=false] [txnStartTS=415022105965101060]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.834 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d44423a3300000000fb00000000000000685449443a35000000fc] [nextEndKey=6d44423a3300000000fb0000000000000069] [reverse=false] [txnStartTS=415022105965101060]
[2020-03-02T21:14:52.844Z] [2020/03/03 05:14:52.835 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d44423a3300000000fb00000000000000685461626c653a3333ff0000000000000000f7] [nextEndKey=6d44423a3300000000fb0000000000000069] [reverse=false] [txnStartTS=415022105965101060]
[2020-03-02T21:14:53.128Z] [2020/03/03 05:14:52.844 +08:00] [INFO] [domain.go:144] ["full load InfoSchema success"] [usedSchemaVersion=0] [neededSchemaVersion=326] ["start time"=14.180188ms]
[2020-03-02T21:14:53.128Z] [2020/03/03 05:14:52.846 +08:00] [INFO] [domain.go:368] ["full load and reset schema validator"]
[2020-03-02T21:14:53.128Z] [2020/03/03 05:14:52.846 +08:00] [DEBUG] [schema_validator.go:139] ["update schema validator"] [oldVer=0] [currVer=326] [changedTableIDs="[]"]
[2020-03-02T21:14:53.128Z] [2020/03/03 05:14:52.883 +08:00] [INFO] [set.go:201] ["set session var"] [conn=0] [name=sql_mode] [val=]
[2020-03-02T21:14:53.128Z] [2020/03/03 05:14:52.883 +08:00] [INFO] [client.go:118] ["load backupmeta"] [databases=1] [jobs=3]
[2020-03-02T21:14:53.128Z] [2020/03/03 05:14:52.883 +08:00] [INFO] [session.go:2071] ["CRUCIAL OPERATION"] [conn=0] [schemaVersion=326] [cur_db=] [sql="CREATE DATABASE /*!32312 IF NOT EXISTS*/ `br_incremental_index` /*!40100 DEFAULT CHARACTER SET utf8mb4 */"] [user=]
[2020-03-02T21:14:53.128Z] [2020/03/03 05:14:52.894 +08:00] [INFO] [collector.go:165] ["Table restore summary: total restore tables: 0, total success: 0, total failed: 0, total take(s): 0.00"]
[2020-03-02T21:14:53.128Z] [2020/03/03 05:14:52.894 +08:00] [INFO] [client.go:101] ["Restore client closed"]
[2020-03-02T21:14:53.128Z] [2020/03/03 05:14:52.894 +08:00] [INFO] [ddl_worker.go:124] ["[ddl] DDL worker closed"] [worker="worker 1, tp general"] ["take time"=10.083µs]
[2020-03-02T21:14:53.128Z] [2020/03/03 05:14:52.895 +08:00] [INFO] [ddl_worker.go:124] ["[ddl] DDL worker closed"] [worker="worker 2, tp add index"] ["take time"=21.664µs]
[2020-03-02T21:14:53.129Z] [2020/03/03 05:14:52.895 +08:00] [INFO] [delete_range.go:123] ["[ddl] closing delRange"]
[2020-03-02T21:14:53.129Z] [2020/03/03 05:14:52.895 +08:00] [INFO] [session_pool.go:85] ["[ddl] closing sessionPool"]
[2020-03-02T21:14:53.129Z] [2020/03/03 05:14:52.895 +08:00] [INFO] [ddl.go:518] ["[ddl] DDL closed"] [ID=005bb9b8-cb65-4aca-b8ee-85b6afdc2ddf] ["take time"=750.978µs]
[2020-03-02T21:14:53.129Z] [2020/03/03 05:14:52.895 +08:00] [INFO] [ddl.go:427] ["[ddl] stop DDL"] [ID=005bb9b8-cb65-4aca-b8ee-85b6afdc2ddf]
[2020-03-02T21:14:53.129Z] [2020/03/03 05:14:52.895 +08:00] [INFO] [manager.go:267] ["failed to campaign"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager 005bb9b8-cb65-4aca-b8ee-85b6afdc2ddf"] [error="context canceled"]
[2020-03-02T21:14:53.129Z] [2020/03/03 05:14:52.895 +08:00] [INFO] [manager.go:248] ["break campaign loop, context is done"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager 005bb9b8-cb65-4aca-b8ee-85b6afdc2ddf"]
[2020-03-02T21:14:53.129Z] [2020/03/03 05:14:52.895 +08:00] [INFO] [manager.go:292] ["revoke session"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager 005bb9b8-cb65-4aca-b8ee-85b6afdc2ddf"] []
[2020-03-02T21:14:53.129Z] [2020/03/03 05:14:52.896 +08:00] [INFO] [domain.go:582] ["domain closed"] ["take time"=2.110551ms]
[2020-03-02T21:14:53.129Z] {"level":"warn","ts":"2020-03-03T05:14:52.896+0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-b5f878f7-74ca-437e-a99b-5ba10a51459c/127.0.0.1:2379","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
[2020-03-02T21:14:53.129Z] [2020/03/03 05:14:52.896 +08:00] [INFO] [manager.go:267] ["failed to campaign"] ["owner info"="[bindinfo] /tidb/bindinfo/owner ownerManager 005bb9b8-cb65-4aca-b8ee-85b6afdc2ddf"] [error="context canceled"]
[2020-03-02T21:14:53.129Z] [2020/03/03 05:14:52.896 +08:00] [INFO] [manager.go:239] ["etcd session is done, creates a new one"] ["owner info"="[bindinfo] /tidb/bindinfo/owner ownerManager 005bb9b8-cb65-4aca-b8ee-85b6afdc2ddf"]
[2020-03-02T21:14:53.129Z] [2020/03/03 05:14:52.896 +08:00] [INFO] [manager.go:243] ["break campaign loop, NewSession failed"] ["owner info"="[bindinfo] /tidb/bindinfo/owner ownerManager 005bb9b8-cb65-4aca-b8ee-85b6afdc2ddf"] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\ngithub.com/pingcap/tidb/owner.contextDone\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200223044457-aedea3ec5e1e/owner/manager.go:371\ngithub.com/pingcap/tidb/owner.NewSession\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200223044457-aedea3ec5e1e/owner/manager.go:142\ngithub.com/pingcap/tidb/owner.(*ownerManager).campaignLoop\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200223044457-aedea3ec5e1e/owner/manager.go:241\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
[2020-03-02T21:14:53.129Z] {"level":"warn","ts":"2020-03-03T05:14:52.896+0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-b5f878f7-74ca-437e-a99b-5ba10a51459c/127.0.0.1:2379","attempt":0,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
[2020-03-02T21:14:53.129Z] Error: all files are filtered out from the backup archive, nothing to restore
[2020-03-02T21:14:53.129Z] all files are filtered out from the backup archive, nothing to restore
[2020-03-02T21:14:53.129Z] github.com/pingcap/br/pkg/task.RunRestore
[2020-03-02T21:14:53.129Z] 	/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/task/restore.go:96
[2020-03-02T21:14:53.129Z] github.com/pingcap/br/cmd.runRestoreCommand
[2020-03-02T21:14:53.129Z] 	/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:17
[2020-03-02T21:14:53.129Z] github.com/pingcap/br/cmd.newTableRestoreCommand.func1
[2020-03-02T21:14:53.129Z] 	/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:77
[2020-03-02T21:14:53.129Z] github.com/spf13/cobra.(*Command).execute
[2020-03-02T21:14:53.129Z] 	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826
[2020-03-02T21:14:53.129Z] github.com/spf13/cobra.(*Command).ExecuteC
[2020-03-02T21:14:53.129Z] 	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
[2020-03-02T21:14:53.129Z] github.com/spf13/cobra.(*Command).Execute
[2020-03-02T21:14:53.129Z] 	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
[2020-03-02T21:14:53.129Z] github.com/pingcap/br.main
[2020-03-02T21:14:53.129Z] 	/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main.go:55
[2020-03-02T21:14:53.129Z] github.com/pingcap/br.TestRunMain.func1
[2020-03-02T21:14:53.129Z] 	/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main_test.go:39
[2020-03-02T21:14:53.129Z] runtime.goexit
[2020-03-02T21:14:53.129Z] 	/usr/local/go/src/runtime/asm_amd64.s:1357

@kennytm
Copy link
Collaborator Author

kennytm commented Mar 2, 2020

/run-integration-test

@codecov
Copy link

codecov bot commented Mar 3, 2020

Codecov Report

Merging #174 into master will not change coverage by %.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #174   +/-   ##
=======================================
  Coverage   71.01%   71.01%           
=======================================
  Files          45       45           
  Lines        4178     4178           
=======================================
  Hits         2967     2967           
  Misses        818      818           
  Partials      393      393           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b88e51...0d06101. Read the comment docs.

@kennytm kennytm requested a review from 3pointer March 3, 2020 09:13
Copy link
Member

@overvenus overvenus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please solve run_services conflicts

@kennytm
Copy link
Collaborator Author

kennytm commented Mar 6, 2020

/run-integration-test

@kennytm kennytm added the status/LGT1 LGTM1 label Mar 10, 2020
@3pointer 3pointer added status/LGT2 LGTM2 and removed status/LGT1 LGTM1 labels Mar 16, 2020
Copy link
Collaborator

@3pointer 3pointer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@overvenus
Copy link
Member

/merge

@sre-bot
Copy link
Contributor

sre-bot commented Mar 16, 2020

/run-all-tests

@sre-bot sre-bot merged commit 156f003 into pingcap:master Mar 16, 2020
kennytm added a commit that referenced this pull request Mar 20, 2020
* restore: merge tidb-tools/pkg/restore-util (#146)

* restore-util: Implement split/scatter (#274)

* implement split/scatter

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* init test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* redesign output/input of the lib

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add commments and more tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add ScanRegions interface to Client

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix potential data race

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore-util/client.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* resolve conflicts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix prefix rewrite

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add RewriteRule/skip failed scatter region/retry the SplitRegion

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check if region has peer

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add split retry interval (#277)

* reset dependencies to release-3.1

* add split retry interval

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix go.sum

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: wait for scatter region sequentially  (#279)

* wait for scatter region sequentially

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add on split hook (#281)

* restore-util: add on split hook

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Nil check onSplit

Co-Authored-By: kennytm <kennytm@gmail.com>

* restore-util: fix returned new region is nil (#283)

* restore-util: fix returned new region is nil

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* *: gofmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: call onSplit on splitByRewriteRules (#285)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore-util: fix overlapped error message (#293)

* restore-util: fix overlapped error message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce error trace

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: log warning when cannot find matched rewrite rule (#299)

* restore-util: add method to set placement rules and store labels (#301)

* restore-util: add method to set placement rules and store labels

Signed-off-by: disksing <i@disksing.com>

* minor fix

Signed-off-by: disksing <i@disksing.com>

* address comment

Signed-off-by: disksing <i@disksing.com>

* add GetPlacementRules

Signed-off-by: disksing <i@disksing.com>

* fix test

Signed-off-by: disksing <i@disksing.com>

* restore-util: support batch split (#300)

* restore-util: support batch split

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* go fmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* address commits

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/restore-util/split.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* add onSplit callback

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add upper bound time for waiting for scatter (#305)

* restore: fix scatter regions failed

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* stop waiting for scatter after 3min

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: fix wrong url (#306)

Signed-off-by: disksing <i@disksing.com>

* restore-util: add warning about unmatched table id (#313)

* restore-util: support table partition

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* warn table id does not match

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* fix compile error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>

* *: prune tidb-tools

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: address linters suggestions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: merge restoreutil into restore

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comment

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>

* Fixed handling for a dbName that do not exist in the backup being restored (#148)

* Fixed handling for a dbName that do not exist in the backup being restored

* Fixed handling for a dbName that do not exist in the backup being restored

* validate: fix debug meta test ci (#153)

* validate: fix debug meta test ci

* *: extracts runBackup/runRestore in cmd into pkg/task (#156)

* *: extracts runBackup/runRestore in cmd into pkg/task

Defines a "Config" structure to store the parsed flags.

Use the "black-white-list" structure to define what tables/databases to
backup/restore.

* go.mod: update tidb to v4.0.0-beta

* restore: fix restore summary log (#150)

Co-authored-by: kennytm <kennytm@gmail.com>

* restore: enhance error handling (#152)

* restore: enhance error handling

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* unit test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix region epoch error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* remove `Restore*`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add debug log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore/import.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix retry error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* handle RegionNotFound error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* Incremental BR: support DDL (#155)

* support backup&restore ddl

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* integration tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update kvproto

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix integration tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce cyclomatic complexity of `runRestore`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable fast checksum in incremental br

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix no valid key error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Reduce TiDB dependencies (#158)

* utils: exclude mock_cluster outside of unit test

* utils: remove unused ResultSetToStringSlice()

* *: abstract away dependencies of tidb/session into a Glue interface

* *: fix hound lint

* util,mock: move utils.MockCluster to mock.Cluster

* restore: fix test build failure

Co-authored-by: 3pointer <qdlc2010@gmail.com>

* go.mod: update tidb (#168)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* BR support TLS (#161)

* *: support tls

* move tikv.driver to glue

* fix comments

* upgrade golangci and prepare for go 1.14 (#171)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* backup: add raw backup command (#101)

* backup: add raw backup command

* restore: speed up retry on not leader (#179)

* tests: stable cluster start up

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: fix unbound var

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: speed retry on not leader

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: add --cacert flag

Signed-off-by: Neil Shen <overvenus@gmail.com>

* make codecov green

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* conn, restore: paginate scan regions (#165)

* conn, restore: paginate scan regions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: large timeout

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Batch restore (#167)

* *: unify Range and RangeTree

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: split restore files into small batch

Signed-off-by: Neil Shen <overvenus@gmail.com>

* task: set default restore concurrency to 128

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: unused table worker pool

Signed-off-by: Neil Shen <overvenus@gmail.com>

* summary: sum up repeated duration and int

Signed-off-by: Neil Shen <overvenus@gmail.com>

* rtree: move rtree from utils to pkg

Signed-off-by: Neil Shen <overvenus@gmail.com>

* README, docker: add quick start (#181)

* README, docker: add quick start

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: disable some TiDB log

Signed-off-by: Neil Shen <overvenus@gmail.com>

* docker: build go-ycsb automatically

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: add TODO about TiDB logs

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: update tidb dependency build with go1.14 (#176)

* *: add license header (#182)

* rtree: move checkFile into backup

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: add license header

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Update LICENSE.md

Co-Authored-By: kennytm <kennytm@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>

* conn: support not shutting down the storage when closing the connection (#185)

Co-authored-by: 3pointer <luancheng@pingcap.com>

* conn: use GetDomain to avoid some TiDB breaking changes (#186)

* conn: use GetDomain to avoid some TiDB breaking changes

Signed-off-by: Neil Shen <overvenus@gmail.com>

* minor usability improvement

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* fix check safepoint & unhide experimental features (#175)

* backup: check safepoint for last backup ts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check lastbackupts > 0

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* unhide experimental features

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update tests/br_z_gc_safepoint/run.sh

Co-Authored-By: kennytm <kennytm@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* support backupts (#172)

* support backupts

* address comment

* address comment

* fix space

* *: update pd deps to v4 (#184)

Co-authored-by: 3pointer <luancheng@pingcap.com>

* restore: support online restore (#114)

Signed-off-by: disksing <i@disksing.com>

* metrics: add grafana scripts (#140)

* add grafana scripts

* fix

Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: glorv <glorvs@163.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* filter out all TiFlash nodes when retrieving lists of stores from PD (#187)

* conn: ignore nodes with label engine=tiflash

* conn: disallow TiFlash on restore, only skip TiFlash on backup

* Create integration test for S3 storage (#174)

* Fix summary log (#191)

* *: fix restore summary log after restore logic changed to files

* fix

* fix

* fix

Co-authored-by: kennytm <kennytm@gmail.com>

* Implement Raw Restore (#104)

* Update kvproto

* Implement raw restore

* fix build

* Set range for file importer

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Remove unnecessary comments

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* check cf and support multi ranges in BackupMeta

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Check files' cf; address comments

* adjust structure to keep consistent with master

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build and make check, avoid accessing TiDB in rawkv mode

* Fix test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix tests

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix broken logic after merging master

* Update pkg/task/restore_raw.go

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* Address comments

* Address comments

* Mark raw restore as experimental

* Fix build

* Address comments

* test: Add check for deleting data and partial backup

* Fix build

* Add license header

* fix ci

* fix ci

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com>

* restore: remove tiflash replica before restore (#194)

* restore: remove tiflash replica before restore

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* rename errSplit variable

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check replica count by region info

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* cleanup

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* save tiflash replica count to backupmeta

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix save crcxor

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix decode the key of placement rule

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* close domain after restoring tiflash-replica

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/task/restore.go

Co-Authored-By: 3pointer <qdlc2010@gmail.com>

Co-authored-by: 3pointer <qdlc2010@gmail.com>

* summary: put summary log at last (#197)

* summary: put summary log at last

* fix switch sql

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Kolbe Kegel <kolbe@pingcap.com>
Co-authored-by: WangXiangUSTC <wx347249478@gmail.com>
Co-authored-by: glorv <glorvs@163.com>
Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com>
3pointer added a commit that referenced this pull request Apr 2, 2020
* restore: fix alter auto increment id for no-primary-key table (#139)

* restore: fix alter auto increment id for no-primary-key table

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add integration test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* *: use oracle package to manipulate ts and test gc safe point  (#121)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: convert version command to flags (#144)

* cmd: convert version command to flags

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: add changelog and github templates (#143)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: merge tidb-tools/pkg/restore-util (#146)

* restore-util: Implement split/scatter (#274)

* implement split/scatter

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* init test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* redesign output/input of the lib

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add commments and more tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add ScanRegions interface to Client

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix potential data race

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore-util/client.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* resolve conflicts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix prefix rewrite

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add RewriteRule/skip failed scatter region/retry the SplitRegion

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check if region has peer

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add split retry interval (#277)

* reset dependencies to release-3.1

* add split retry interval

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix go.sum

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: wait for scatter region sequentially  (#279)

* wait for scatter region sequentially

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add on split hook (#281)

* restore-util: add on split hook

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Nil check onSplit

Co-Authored-By: kennytm <kennytm@gmail.com>

* restore-util: fix returned new region is nil (#283)

* restore-util: fix returned new region is nil

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* *: gofmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: call onSplit on splitByRewriteRules (#285)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore-util: fix overlapped error message (#293)

* restore-util: fix overlapped error message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce error trace

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: log warning when cannot find matched rewrite rule (#299)

* restore-util: add method to set placement rules and store labels (#301)

* restore-util: add method to set placement rules and store labels

Signed-off-by: disksing <i@disksing.com>

* minor fix

Signed-off-by: disksing <i@disksing.com>

* address comment

Signed-off-by: disksing <i@disksing.com>

* add GetPlacementRules

Signed-off-by: disksing <i@disksing.com>

* fix test

Signed-off-by: disksing <i@disksing.com>

* restore-util: support batch split (#300)

* restore-util: support batch split

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* go fmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* address commits

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/restore-util/split.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* add onSplit callback

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add upper bound time for waiting for scatter (#305)

* restore: fix scatter regions failed

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* stop waiting for scatter after 3min

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: fix wrong url (#306)

Signed-off-by: disksing <i@disksing.com>

* restore-util: add warning about unmatched table id (#313)

* restore-util: support table partition

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* warn table id does not match

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* fix compile error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>

* *: prune tidb-tools

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: address linters suggestions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: merge restoreutil into restore

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comment

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>

* Fixed handling for a dbName that do not exist in the backup being restored (#148)

* Fixed handling for a dbName that do not exist in the backup being restored

* Fixed handling for a dbName that do not exist in the backup being restored

* validate: fix debug meta test ci (#153)

* validate: fix debug meta test ci

* *: extracts runBackup/runRestore in cmd into pkg/task (#156)

* *: extracts runBackup/runRestore in cmd into pkg/task

Defines a "Config" structure to store the parsed flags.

Use the "black-white-list" structure to define what tables/databases to
backup/restore.

* go.mod: update tidb to v4.0.0-beta

* restore: fix restore summary log (#150)

Co-authored-by: kennytm <kennytm@gmail.com>

* restore: enhance error handling (#152)

* restore: enhance error handling

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* unit test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix region epoch error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* remove `Restore*`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add debug log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore/import.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix retry error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* handle RegionNotFound error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* Incremental BR: support DDL (#155)

* support backup&restore ddl

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* integration tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update kvproto

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix integration tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce cyclomatic complexity of `runRestore`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable fast checksum in incremental br

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix no valid key error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Reduce TiDB dependencies (#158)

* utils: exclude mock_cluster outside of unit test

* utils: remove unused ResultSetToStringSlice()

* *: abstract away dependencies of tidb/session into a Glue interface

* *: fix hound lint

* util,mock: move utils.MockCluster to mock.Cluster

* restore: fix test build failure

Co-authored-by: 3pointer <qdlc2010@gmail.com>

* go.mod: update tidb (#168)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* BR support TLS (#161)

* *: support tls

* move tikv.driver to glue

* fix comments

* upgrade golangci and prepare for go 1.14 (#171)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* backup: add raw backup command (#101)

* backup: add raw backup command

* restore: speed up retry on not leader (#179)

* tests: stable cluster start up

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: fix unbound var

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: speed retry on not leader

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: add --cacert flag

Signed-off-by: Neil Shen <overvenus@gmail.com>

* make codecov green

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* conn, restore: paginate scan regions (#165)

* conn, restore: paginate scan regions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: large timeout

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Batch restore (#167)

* *: unify Range and RangeTree

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: split restore files into small batch

Signed-off-by: Neil Shen <overvenus@gmail.com>

* task: set default restore concurrency to 128

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: unused table worker pool

Signed-off-by: Neil Shen <overvenus@gmail.com>

* summary: sum up repeated duration and int

Signed-off-by: Neil Shen <overvenus@gmail.com>

* rtree: move rtree from utils to pkg

Signed-off-by: Neil Shen <overvenus@gmail.com>

* README, docker: add quick start (#181)

* README, docker: add quick start

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: disable some TiDB log

Signed-off-by: Neil Shen <overvenus@gmail.com>

* docker: build go-ycsb automatically

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: add TODO about TiDB logs

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: update tidb dependency build with go1.14 (#176)

* *: add license header (#182)

* rtree: move checkFile into backup

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: add license header

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Update LICENSE.md

Co-Authored-By: kennytm <kennytm@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>

* conn: support not shutting down the storage when closing the connection (#185)

Co-authored-by: 3pointer <luancheng@pingcap.com>

* conn: use GetDomain to avoid some TiDB breaking changes (#186)

* conn: use GetDomain to avoid some TiDB breaking changes

Signed-off-by: Neil Shen <overvenus@gmail.com>

* minor usability improvement

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* fix check safepoint & unhide experimental features (#175)

* backup: check safepoint for last backup ts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check lastbackupts > 0

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* unhide experimental features

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update tests/br_z_gc_safepoint/run.sh

Co-Authored-By: kennytm <kennytm@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* support backupts (#172)

* support backupts

* address comment

* address comment

* fix space

* *: update pd deps to v4 (#184)

Co-authored-by: 3pointer <luancheng@pingcap.com>

* restore: support online restore (#114)

Signed-off-by: disksing <i@disksing.com>

* metrics: add grafana scripts (#140)

* add grafana scripts

* fix

Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: glorv <glorvs@163.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* filter out all TiFlash nodes when retrieving lists of stores from PD (#187)

* conn: ignore nodes with label engine=tiflash

* conn: disallow TiFlash on restore, only skip TiFlash on backup

* Create integration test for S3 storage (#174)

* Fix summary log (#191)

* *: fix restore summary log after restore logic changed to files

* fix

* fix

* fix

Co-authored-by: kennytm <kennytm@gmail.com>

* Implement Raw Restore (#104)

* Update kvproto

* Implement raw restore

* fix build

* Set range for file importer

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Remove unnecessary comments

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* check cf and support multi ranges in BackupMeta

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Check files' cf; address comments

* adjust structure to keep consistent with master

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build and make check, avoid accessing TiDB in rawkv mode

* Fix test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix tests

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix broken logic after merging master

* Update pkg/task/restore_raw.go

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* Address comments

* Address comments

* Mark raw restore as experimental

* Fix build

* Address comments

* test: Add check for deleting data and partial backup

* Fix build

* Add license header

* fix ci

* fix ci

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com>

* update dep

* restore: remove tiflash replica before restore (#194)

* restore: remove tiflash replica before restore

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* rename errSplit variable

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check replica count by region info

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* cleanup

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* save tiflash replica count to backupmeta

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix save crcxor

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix decode the key of placement rule

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* close domain after restoring tiflash-replica

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/task/restore.go

Co-Authored-By: 3pointer <qdlc2010@gmail.com>

Co-authored-by: 3pointer <qdlc2010@gmail.com>

* summary: put summary log at last (#197)

* summary: put summary log at last

* fix switch sql

* *: abstract the progress channel (updateCh) into the glue package (#196)

* *: abstract the progress channel (updateCh) into the glue package

* restore: fix crash in truncateTS() when the bound is unlimited

* task: fix comment

Co-authored-by: Ian <ArGregoryIan@gmail.com>

* *: refline logs (#189)

* adapt pd/tidb related 3.1 changes

* tests: disable TLS test (#204)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: add S3 quick start and few enhancement of log (#202)

* README, docker: add quick start about S3 storage

Signed-off-by: Neil Shen <overvenus@gmail.com>

* pkg/summary: make sure to output correct summary

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd, tests: log to terminal if BR_LOG_TO_TERM is set

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Update pkg/task/common.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: cat log if br fails

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* restore: add error field to `DownloadResponse` (#195)

* restore: add error field to `DownloadResponse`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore: populate restore cancel error (#207)

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* enhance usability of br (#208)

* silenceUsage only when parse cmd flags failed

* udpate tidb

Co-authored-by: kennytm <kennytm@gmail.com>

* task: do not run checksum if restore failed (#209)

* fix incremental bug in llroad test (#199)

* restore: filter same table ddl

* *: do not return error when backup/restore data is empty

* fix create database double during incremental restore

* add tests

* fix ci

* address comment

* add skip create sqls (#211)

* Revert "tests: disable TLS test (#204)" (#218)

This reverts commit e168a60.

* doc: add `minio` to dependence list. (#221)

The README of test omitted `minio` in the dependence list, which is needed for run the integration test.

Co-authored-by: Neil Shen <overvenus@gmail.com>

* move waiting reject stores in import file (#222)

* move wait rejectstores into import files

* restore: use new table id to search placementRules

* Update pkg/restore/import.go

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* Update pkg/restore/import.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix ci

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* Max index length (#220)

* restore: set max-index-length to max

* restore:add max-index-length params

* address comment

* address comment

* glue: create schema/table directly with info (#216)

* glue: create schema/table directly with info

* go.mod: change to use the master version

* gluetidb: fix failure to create schema

* gluetidb: exclude non-public indices when restoring

* go.mod: removed unused replace

Co-authored-by: 3pointer <luancheng@pingcap.com>

* update deps

Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Kolbe Kegel <kolbe@pingcap.com>
Co-authored-by: WangXiangUSTC <wx347249478@gmail.com>
Co-authored-by: glorv <glorvs@163.com>
Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com>
Co-authored-by: 庄天翼 <zty0826@gmail.com>
Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
3pointer added a commit that referenced this pull request Apr 2, 2020
* restore: merge tidb-tools/pkg/restore-util (#146)

* restore-util: Implement split/scatter (#274)

* implement split/scatter

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* init test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* redesign output/input of the lib

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add commments and more tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add ScanRegions interface to Client

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix potential data race

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore-util/client.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* resolve conflicts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix prefix rewrite

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add RewriteRule/skip failed scatter region/retry the SplitRegion

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check if region has peer

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add split retry interval (#277)

* reset dependencies to release-3.1

* add split retry interval

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix go.sum

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: wait for scatter region sequentially  (#279)

* wait for scatter region sequentially

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add on split hook (#281)

* restore-util: add on split hook

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Nil check onSplit

Co-Authored-By: kennytm <kennytm@gmail.com>

* restore-util: fix returned new region is nil (#283)

* restore-util: fix returned new region is nil

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* *: gofmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: call onSplit on splitByRewriteRules (#285)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore-util: fix overlapped error message (#293)

* restore-util: fix overlapped error message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce error trace

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: log warning when cannot find matched rewrite rule (#299)

* restore-util: add method to set placement rules and store labels (#301)

* restore-util: add method to set placement rules and store labels

Signed-off-by: disksing <i@disksing.com>

* minor fix

Signed-off-by: disksing <i@disksing.com>

* address comment

Signed-off-by: disksing <i@disksing.com>

* add GetPlacementRules

Signed-off-by: disksing <i@disksing.com>

* fix test

Signed-off-by: disksing <i@disksing.com>

* restore-util: support batch split (#300)

* restore-util: support batch split

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* go fmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* address commits

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/restore-util/split.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* add onSplit callback

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add upper bound time for waiting for scatter (#305)

* restore: fix scatter regions failed

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* stop waiting for scatter after 3min

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: fix wrong url (#306)

Signed-off-by: disksing <i@disksing.com>

* restore-util: add warning about unmatched table id (#313)

* restore-util: support table partition

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* warn table id does not match

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* fix compile error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>

* *: prune tidb-tools

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: address linters suggestions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: merge restoreutil into restore

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comment

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>

* Fixed handling for a dbName that do not exist in the backup being restored (#148)

* Fixed handling for a dbName that do not exist in the backup being restored

* Fixed handling for a dbName that do not exist in the backup being restored

* validate: fix debug meta test ci (#153)

* validate: fix debug meta test ci

* *: extracts runBackup/runRestore in cmd into pkg/task (#156)

* *: extracts runBackup/runRestore in cmd into pkg/task

Defines a "Config" structure to store the parsed flags.

Use the "black-white-list" structure to define what tables/databases to
backup/restore.

* go.mod: update tidb to v4.0.0-beta

* restore: fix restore summary log (#150)

Co-authored-by: kennytm <kennytm@gmail.com>

* restore: enhance error handling (#152)

* restore: enhance error handling

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* unit test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix region epoch error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* remove `Restore*`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add debug log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore/import.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix retry error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* handle RegionNotFound error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* Incremental BR: support DDL (#155)

* support backup&restore ddl

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* integration tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update kvproto

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix integration tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce cyclomatic complexity of `runRestore`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable fast checksum in incremental br

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix no valid key error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Reduce TiDB dependencies (#158)

* utils: exclude mock_cluster outside of unit test

* utils: remove unused ResultSetToStringSlice()

* *: abstract away dependencies of tidb/session into a Glue interface

* *: fix hound lint

* util,mock: move utils.MockCluster to mock.Cluster

* restore: fix test build failure

Co-authored-by: 3pointer <qdlc2010@gmail.com>

* go.mod: update tidb (#168)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* BR support TLS (#161)

* *: support tls

* move tikv.driver to glue

* fix comments

* upgrade golangci and prepare for go 1.14 (#171)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* backup: add raw backup command (#101)

* backup: add raw backup command

* restore: speed up retry on not leader (#179)

* tests: stable cluster start up

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: fix unbound var

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: speed retry on not leader

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: add --cacert flag

Signed-off-by: Neil Shen <overvenus@gmail.com>

* make codecov green

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* conn, restore: paginate scan regions (#165)

* conn, restore: paginate scan regions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: large timeout

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Batch restore (#167)

* *: unify Range and RangeTree

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: split restore files into small batch

Signed-off-by: Neil Shen <overvenus@gmail.com>

* task: set default restore concurrency to 128

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: unused table worker pool

Signed-off-by: Neil Shen <overvenus@gmail.com>

* summary: sum up repeated duration and int

Signed-off-by: Neil Shen <overvenus@gmail.com>

* rtree: move rtree from utils to pkg

Signed-off-by: Neil Shen <overvenus@gmail.com>

* README, docker: add quick start (#181)

* README, docker: add quick start

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: disable some TiDB log

Signed-off-by: Neil Shen <overvenus@gmail.com>

* docker: build go-ycsb automatically

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: add TODO about TiDB logs

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: update tidb dependency build with go1.14 (#176)

* *: add license header (#182)

* rtree: move checkFile into backup

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: add license header

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Update LICENSE.md

Co-Authored-By: kennytm <kennytm@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>

* conn: support not shutting down the storage when closing the connection (#185)

Co-authored-by: 3pointer <luancheng@pingcap.com>

* conn: use GetDomain to avoid some TiDB breaking changes (#186)

* conn: use GetDomain to avoid some TiDB breaking changes

Signed-off-by: Neil Shen <overvenus@gmail.com>

* minor usability improvement

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* fix check safepoint & unhide experimental features (#175)

* backup: check safepoint for last backup ts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check lastbackupts > 0

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* unhide experimental features

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update tests/br_z_gc_safepoint/run.sh

Co-Authored-By: kennytm <kennytm@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* support backupts (#172)

* support backupts

* address comment

* address comment

* fix space

* *: update pd deps to v4 (#184)

Co-authored-by: 3pointer <luancheng@pingcap.com>

* restore: support online restore (#114)

Signed-off-by: disksing <i@disksing.com>

* metrics: add grafana scripts (#140)

* add grafana scripts

* fix

Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: glorv <glorvs@163.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* filter out all TiFlash nodes when retrieving lists of stores from PD (#187)

* conn: ignore nodes with label engine=tiflash

* conn: disallow TiFlash on restore, only skip TiFlash on backup

* Create integration test for S3 storage (#174)

* Fix summary log (#191)

* *: fix restore summary log after restore logic changed to files

* fix

* fix

* fix

Co-authored-by: kennytm <kennytm@gmail.com>

* Implement Raw Restore (#104)

* Update kvproto

* Implement raw restore

* fix build

* Set range for file importer

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Remove unnecessary comments

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* check cf and support multi ranges in BackupMeta

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Check files' cf; address comments

* adjust structure to keep consistent with master

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build and make check, avoid accessing TiDB in rawkv mode

* Fix test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix tests

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix broken logic after merging master

* Update pkg/task/restore_raw.go

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* Address comments

* Address comments

* Mark raw restore as experimental

* Fix build

* Address comments

* test: Add check for deleting data and partial backup

* Fix build

* Add license header

* fix ci

* fix ci

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com>

* restore: remove tiflash replica before restore (#194)

* restore: remove tiflash replica before restore

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* rename errSplit variable

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check replica count by region info

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* cleanup

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* save tiflash replica count to backupmeta

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix save crcxor

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix decode the key of placement rule

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* close domain after restoring tiflash-replica

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/task/restore.go

Co-Authored-By: 3pointer <qdlc2010@gmail.com>

Co-authored-by: 3pointer <qdlc2010@gmail.com>

* summary: put summary log at last (#197)

* summary: put summary log at last

* fix switch sql

* *: abstract the progress channel (updateCh) into the glue package (#196)

* *: abstract the progress channel (updateCh) into the glue package

* restore: fix crash in truncateTS() when the bound is unlimited

* task: fix comment

Co-authored-by: Ian <ArGregoryIan@gmail.com>

* *: refline logs (#189)

* tests: disable TLS test (#204)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: add S3 quick start and few enhancement of log (#202)

* README, docker: add quick start about S3 storage

Signed-off-by: Neil Shen <overvenus@gmail.com>

* pkg/summary: make sure to output correct summary

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd, tests: log to terminal if BR_LOG_TO_TERM is set

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Update pkg/task/common.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: cat log if br fails

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* restore: add error field to `DownloadResponse` (#195)

* restore: add error field to `DownloadResponse`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore: populate restore cancel error (#207)

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* enhance usability of br (#208)

* silenceUsage only when parse cmd flags failed

* udpate tidb

Co-authored-by: kennytm <kennytm@gmail.com>

* task: do not run checksum if restore failed (#209)

* fix incremental bug in llroad test (#199)

* restore: filter same table ddl

* *: do not return error when backup/restore data is empty

* fix create database double during incremental restore

* add tests

* fix ci

* address comment

* add skip create sqls (#211)

* Revert "tests: disable TLS test (#204)" (#218)

This reverts commit e168a60.

* doc: add `minio` to dependence list. (#221)

The README of test omitted `minio` in the dependence list, which is needed for run the integration test.

Co-authored-by: Neil Shen <overvenus@gmail.com>

* move waiting reject stores in import file (#222)

* move wait rejectstores into import files

* restore: use new table id to search placementRules

* Update pkg/restore/import.go

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* Update pkg/restore/import.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix ci

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* Max index length (#220)

* restore: set max-index-length to max

* restore:add max-index-length params

* address comment

* address comment

* glue: create schema/table directly with info (#216)

* glue: create schema/table directly with info

* go.mod: change to use the master version

* gluetidb: fix failure to create schema

* gluetidb: exclude non-public indices when restoring

* go.mod: removed unused replace

Co-authored-by: 3pointer <luancheng@pingcap.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Kolbe Kegel <kolbe@pingcap.com>
Co-authored-by: WangXiangUSTC <wx347249478@gmail.com>
Co-authored-by: glorv <glorvs@163.com>
Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com>
Co-authored-by: 庄天翼 <zty0826@gmail.com>
Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Integration test for S3
4 participants