Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge master to Release 4.0 (pingcap#206)
* restore: merge tidb-tools/pkg/restore-util (pingcap#146) * restore-util: Implement split/scatter (pingcap#274) * implement split/scatter Signed-off-by: 5kbpers <tangminghua@pingcap.com> * init test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * redesign output/input of the lib Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add commments and more tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add ScanRegions interface to Client Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix potential data race Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * resolve conflicts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix prefix rewrite Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check if region has peer Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add split retry interval (pingcap#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix go.sum Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: wait for scatter region sequentially (pingcap#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add on split hook (pingcap#281) * restore-util: add on split hook Signed-off-by: Neil Shen <overvenus@gmail.com> * Nil check onSplit Co-Authored-By: kennytm <kennytm@gmail.com> * restore-util: fix returned new region is nil (pingcap#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * *: gofmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: call onSplit on splitByRewriteRules (pingcap#285) Signed-off-by: Neil Shen <overvenus@gmail.com> * restore-util: fix overlapped error message (pingcap#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce error trace Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: log warning when cannot find matched rewrite rule (pingcap#299) * restore-util: add method to set placement rules and store labels (pingcap#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <i@disksing.com> * minor fix Signed-off-by: disksing <i@disksing.com> * address comment Signed-off-by: disksing <i@disksing.com> * add GetPlacementRules Signed-off-by: disksing <i@disksing.com> * fix test Signed-off-by: disksing <i@disksing.com> * restore-util: support batch split (pingcap#300) * restore-util: support batch split Signed-off-by: 5kbpers <tangminghua@pingcap.com> * go fmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * address commits Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <kennytm@gmail.com> * add onSplit callback Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add upper bound time for waiting for scatter (pingcap#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: fix wrong url (pingcap#306) Signed-off-by: disksing <i@disksing.com> * restore-util: add warning about unmatched table id (pingcap#313) * restore-util: support table partition Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * warn table id does not match Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: Neil Shen <overvenus@gmail.com> * fix compile error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> * *: prune tidb-tools Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: address linters suggestions Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <overvenus@gmail.com> * address comment Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> * Fixed handling for a dbName that do not exist in the backup being restored (pingcap#148) * Fixed handling for a dbName that do not exist in the backup being restored * Fixed handling for a dbName that do not exist in the backup being restored * validate: fix debug meta test ci (pingcap#153) * validate: fix debug meta test ci * *: extracts runBackup/runRestore in cmd into pkg/task (pingcap#156) * *: extracts runBackup/runRestore in cmd into pkg/task Defines a "Config" structure to store the parsed flags. Use the "black-white-list" structure to define what tables/databases to backup/restore. * go.mod: update tidb to v4.0.0-beta * restore: fix restore summary log (pingcap#150) Co-authored-by: kennytm <kennytm@gmail.com> * restore: enhance error handling (pingcap#152) * restore: enhance error handling Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix region epoch error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * remove `Restore*` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add debug log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore/import.go Co-Authored-By: kennytm <kennytm@gmail.com> * fix retry error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * handle RegionNotFound error Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * Incremental BR: support DDL (pingcap#155) * support backup&restore ddl Signed-off-by: 5kbpers <tangminghua@pingcap.com> * integration tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update kvproto Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix integration tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce cyclomatic complexity of `runRestore` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable fast checksum in incremental br Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix no valid key error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Reduce TiDB dependencies (pingcap#158) * utils: exclude mock_cluster outside of unit test * utils: remove unused ResultSetToStringSlice() * *: abstract away dependencies of tidb/session into a Glue interface * *: fix hound lint * util,mock: move utils.MockCluster to mock.Cluster * restore: fix test build failure Co-authored-by: 3pointer <qdlc2010@gmail.com> * go.mod: update tidb (pingcap#168) Signed-off-by: Neil Shen <overvenus@gmail.com> * BR support TLS (pingcap#161) * *: support tls * move tikv.driver to glue * fix comments * upgrade golangci and prepare for go 1.14 (pingcap#171) Signed-off-by: Neil Shen <overvenus@gmail.com> * backup: add raw backup command (pingcap#101) * backup: add raw backup command * restore: speed up retry on not leader (pingcap#179) * tests: stable cluster start up Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: fix unbound var Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: speed retry on not leader Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: add --cacert flag Signed-off-by: Neil Shen <overvenus@gmail.com> * make codecov green Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * conn, restore: paginate scan regions (pingcap#165) * conn, restore: paginate scan regions Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: large timeout Signed-off-by: Neil Shen <overvenus@gmail.com> * Batch restore (pingcap#167) * *: unify Range and RangeTree Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: split restore files into small batch Signed-off-by: Neil Shen <overvenus@gmail.com> * task: set default restore concurrency to 128 Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: unused table worker pool Signed-off-by: Neil Shen <overvenus@gmail.com> * summary: sum up repeated duration and int Signed-off-by: Neil Shen <overvenus@gmail.com> * rtree: move rtree from utils to pkg Signed-off-by: Neil Shen <overvenus@gmail.com> * README, docker: add quick start (pingcap#181) * README, docker: add quick start Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: disable some TiDB log Signed-off-by: Neil Shen <overvenus@gmail.com> * docker: build go-ycsb automatically Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: add TODO about TiDB logs Signed-off-by: Neil Shen <overvenus@gmail.com> * *: update tidb dependency build with go1.14 (pingcap#176) * *: add license header (pingcap#182) * rtree: move checkFile into backup Signed-off-by: Neil Shen <overvenus@gmail.com> * *: add license header Signed-off-by: Neil Shen <overvenus@gmail.com> * Update LICENSE.md Co-Authored-By: kennytm <kennytm@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: 3pointer <luancheng@pingcap.com> * conn: support not shutting down the storage when closing the connection (pingcap#185) Co-authored-by: 3pointer <luancheng@pingcap.com> * conn: use GetDomain to avoid some TiDB breaking changes (pingcap#186) * conn: use GetDomain to avoid some TiDB breaking changes Signed-off-by: Neil Shen <overvenus@gmail.com> * minor usability improvement Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * fix check safepoint & unhide experimental features (pingcap#175) * backup: check safepoint for last backup ts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check lastbackupts > 0 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unhide experimental features Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update tests/br_z_gc_safepoint/run.sh Co-Authored-By: kennytm <kennytm@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * support backupts (pingcap#172) * support backupts * address comment * address comment * fix space * *: update pd deps to v4 (pingcap#184) Co-authored-by: 3pointer <luancheng@pingcap.com> * restore: support online restore (pingcap#114) Signed-off-by: disksing <i@disksing.com> * metrics: add grafana scripts (pingcap#140) * add grafana scripts * fix Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: kennytm <kennytm@gmail.com> * filter out all TiFlash nodes when retrieving lists of stores from PD (pingcap#187) * conn: ignore nodes with label engine=tiflash * conn: disallow TiFlash on restore, only skip TiFlash on backup * Create integration test for S3 storage (pingcap#174) * Fix summary log (pingcap#191) * *: fix restore summary log after restore logic changed to files * fix * fix * fix Co-authored-by: kennytm <kennytm@gmail.com> * Implement Raw Restore (pingcap#104) * Update kvproto * Implement raw restore * fix build * Set range for file importer Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Remove unnecessary comments Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * check cf and support multi ranges in BackupMeta Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Check files' cf; address comments * adjust structure to keep consistent with master * Fix build Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix build and make check, avoid accessing TiDB in rawkv mode * Fix test Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix tests Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix broken logic after merging master * Update pkg/task/restore_raw.go Co-Authored-By: Neil Shen <overvenus@gmail.com> * Address comments * Address comments * Mark raw restore as experimental * Fix build * Address comments * test: Add check for deleting data and partial backup * Fix build * Add license header * fix ci * fix ci Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com> * restore: remove tiflash replica before restore (pingcap#194) * restore: remove tiflash replica before restore Signed-off-by: 5kbpers <tangminghua@pingcap.com> * rename errSplit variable Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check replica count by region info Signed-off-by: 5kbpers <tangminghua@pingcap.com> * cleanup Signed-off-by: 5kbpers <tangminghua@pingcap.com> * save tiflash replica count to backupmeta Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix save crcxor Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix decode the key of placement rule Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close domain after restoring tiflash-replica Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/task/restore.go Co-Authored-By: 3pointer <qdlc2010@gmail.com> Co-authored-by: 3pointer <qdlc2010@gmail.com> * summary: put summary log at last (pingcap#197) * summary: put summary log at last * fix switch sql * *: abstract the progress channel (updateCh) into the glue package (pingcap#196) * *: abstract the progress channel (updateCh) into the glue package * restore: fix crash in truncateTS() when the bound is unlimited * task: fix comment Co-authored-by: Ian <ArGregoryIan@gmail.com> * *: refline logs (pingcap#189) * tests: disable TLS test (pingcap#204) Signed-off-by: Neil Shen <overvenus@gmail.com> * *: add S3 quick start and few enhancement of log (pingcap#202) * README, docker: add quick start about S3 storage Signed-off-by: Neil Shen <overvenus@gmail.com> * pkg/summary: make sure to output correct summary Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd, tests: log to terminal if BR_LOG_TO_TERM is set Signed-off-by: Neil Shen <overvenus@gmail.com> * Update pkg/task/common.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: cat log if br fails Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * restore: add error field to `DownloadResponse` (pingcap#195) * restore: add error field to `DownloadResponse` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore: populate restore cancel error (pingcap#207) Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * enhance usability of br (pingcap#208) * silenceUsage only when parse cmd flags failed * udpate tidb Co-authored-by: kennytm <kennytm@gmail.com> * task: do not run checksum if restore failed (pingcap#209) * fix incremental bug in llroad test (pingcap#199) * restore: filter same table ddl * *: do not return error when backup/restore data is empty * fix create database double during incremental restore * add tests * fix ci * address comment * add skip create sqls (pingcap#211) * Revert "tests: disable TLS test (pingcap#204)" (pingcap#218) This reverts commit e168a60. * doc: add `minio` to dependence list. (pingcap#221) The README of test omitted `minio` in the dependence list, which is needed for run the integration test. Co-authored-by: Neil Shen <overvenus@gmail.com> * move waiting reject stores in import file (pingcap#222) * move wait rejectstores into import files * restore: use new table id to search placementRules * Update pkg/restore/import.go Co-Authored-By: Neil Shen <overvenus@gmail.com> * Update pkg/restore/import.go Co-Authored-By: kennytm <kennytm@gmail.com> * fix ci Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * Max index length (pingcap#220) * restore: set max-index-length to max * restore:add max-index-length params * address comment * address comment * glue: create schema/table directly with info (pingcap#216) * glue: create schema/table directly with info * go.mod: change to use the master version * gluetidb: fix failure to create schema * gluetidb: exclude non-public indices when restoring * go.mod: removed unused replace Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Kolbe Kegel <kolbe@pingcap.com> Co-authored-by: WangXiangUSTC <wx347249478@gmail.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com> Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com> Co-authored-by: 庄天翼 <zty0826@gmail.com> Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
- Loading branch information
14 people
authored
Apr 2, 2020
1 parent
3239e7d
commit 194a4ff