br restore failed due to "rpc error: code = DeadlineExceeded desc = context deadline exceeded: [BR:PD:ErrPDBatchScanRegion]batch scan region"

[fubinzh]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. A TiDB cluster with 20 TiKVs (16c64g)
2. Use br to restore full for a backup for 500k tables with stats files (cluster size: 30TB)

restore full -s gcs://aaa/bbb-500k-table-30tb-backup-with-stats-3 -u http://tc-pd:2379 --check-requirements=false --ddl-batch-size=128 --tikv-max-restore-concurrency=128 --granularity=coarse-grained --send-credentials-to-tikv=true --checksum=false

2. What did you expect to see? (Required)

Restore should suceeded.

3. What did you see instead (Required)

Restore failed

[root@br-0 tmp]# grep ERROR br.log.2024-01-16T01.32-2024-01-16T05-08*| head -n1
br.log.2024-01-16T01.32-2024-01-16T05-08-30.623.24Z:[2024/01/16 05:08:27.141 +00:00] [ERROR] [import.go:610] ["import sst file failed after retry, stop the whole progress"] [files="{total=1,files=\"[41_4663603_707_3bcdc3254f212dc0fa6cd052ecaa6dcea1a6e223b8707d83c05e922768a62681_1704689362167_write.sst]\",totalKVs=10000,totalBytes=360000,totalSize=41527}"] [error="scan regions from start-key:74800000000004ABFF4F5F698000000000FF0000020000000000FB, err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: [BR:PD:ErrPDBatchScanRegion]batch scan region; scan regions from start-key:74800000000004ABFF4F5F698000000000FF0000020000000000FB, err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: [BR:PD:ErrPDBatchScanRegion]batch scan region; scan regions from start-key:74800000000004ABFF4F5F698000000000FF0000020000000000FB, err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: [BR:PD:ErrPDBatchScanRegion]batch scan region; scan regions from start-key:74800000000004ABFF4F5F698000000000FF0000020000000000FB, err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: [BR:PD:ErrPDBatchScanRegion]batch scan region"] [errorVerbose="the following errors occurred:\n -  [BR:PD:ErrPDBatchScanRegion]batch scan region\n    scan regions from start-key:74800000000004ABFF4F5F698000000000FF0000020000000000FB, err: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n    github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:108\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:213\n    github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:231\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:212\n    github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:101\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:213\n    github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:231\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:212\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:547\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2.1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1506\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1507\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1650\n -  [BR:PD:ErrPDBatchScanRegion]batch scan region\n    scan regions from start-key:74800000000004ABFF4F5F698000000000FF0000020000000000FB, err: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n    github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:108\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:213\n    github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:231\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:212\n    github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:101\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:213\n    github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:231\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:212\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:547\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2.1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1506\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1507\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1650\n -  [BR:PD:ErrPDBatchScanRegion]batch scan region\n    scan regions from start-key:74800000000004ABFF4F5F698000000000FF0000020000000000FB, err: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n    github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:108\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:213\n    github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:231\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:212\n    github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:101\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:213\n    github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:231\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:212\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:547\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2.1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1506\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1507\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1650\n -  [BR:PD:ErrPDBatchScanRegion]batch scan region\n    scan regions from start-key:74800000000004ABFF4F5F698000000000FF0000020000000000FB, err: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n    github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:108\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:213\n    github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:231\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:212\n    github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:101\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:213\n    github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:231\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:212\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:547\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2.1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1506\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1507\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1650"] [stack="github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:610\ngithub.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2.1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1506\ngithub.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1507\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75"]
[root@br-0 tmp]#

4. What is your TiDB version? (Required)

[root@br-0 tmp]# /br -V
Release Version: v7.6.0
Git Commit Hash: 6701bf1
Git Branch: heads/refs/tags/v7.6.0
Go Version: go1.21.5
UTC Build Time: 2024-01-08 12:55:13
Race Enabled: false

[fubinzh]

/component br
/severity critical

[Leavrth]

panic by other error before the really error information output