br restore data from s3, occur [BR:PD:ErrPDBatchScanRegion]batch scan region err: rpc error: code = DeadlineExceeded desc = context deadline exceeded

[River2000i]

Bug Report

[2022/12/03 05:16:47.660 +00:00] [ERROR] [restore.go:59] ["failed to restore"] [error="scan regions from start-key:748000000000000AFFAB5F698000000000FF0000010380000000FF0000022803800000FF0000F4D4AD000000FC, err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: [BR:PD:ErrPDBatchScanRegion]batch scan region"] [errorVerbose="[BR:PD:ErrPDBatchScanRegion]batch scan region
scan regions from start-key:748000000000000AFFAB5F698000000000FF0000010380000000FF0000022803800000FF0000F4D4AD000000FC, err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1
  /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:98
github.com/pingcap/tidb/br/pkg/utils.WithRetry
  /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:52
github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion
  /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:91
github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1
  /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:496
github.com/pingcap/tidb/br/pkg/utils.WithRetry
  /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:52
github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles
  /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:492
github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2
  /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1148
github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1
  /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76
golang.org/x/sync/errgroup.(*Group).Go.func1
  /go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75
runtime.goexit
  /usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.runRestoreCommand
  /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/restore.go:59
main.newFullRestoreCommand.func1
  /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/restore.go:143
github.com/spf13/cobra.(*Command).execute
  /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
  /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044
github.com/spf13/cobra.(*Command).Execute
  /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
main.main
  /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57
runtime.main
  /usr/local/go/src/runtime/proc.go:250"]

What did you do?

br restore data from s3

What did you expect to see?

br restore success

What did you see instead?

br restore failure

What version of PD are you using (pd-server -V)?

./pd-server -V
Release Version: v6.5.0
Edition: Community
Git Commit Hash: 5c1bb89
Git Branch: heads/refs/tags/v6.5.0
UTC Build Time: 2022-12-02 08:48:33

[River2000i]

/severity Moderate
/assign rleungx

[River2000i]

FYI, same backup data restore success in the next day

[rleungx]

Close it since #5758 has been merged. Feel free to reopen it if there is still a problem.