pd incorrectly report "scatter region failed" in normal scatter reigon scenario

[seiya-annie]

Bug Report

What did you do?

open titan
lightning import data to tidb cluster

What did you expect to see?

import successfully

What did you see instead?

[2023/09/08 07:27:00.619 +00:00] [WARN] [localhelper.go:418] ["scatter region failed"] [regionCount=4] [failedCount=2] [error="scatter region 3550 failed: type:UNKNOWN message:\"operator could not be allocated\" "] [errorVerbose="scatter region 3550 failed: type:UNKNOWN message:\"operator could not be allocated\" \[ngithub.com/tikv/pd/client.(*client).scatterRegionsWithGroup\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20230728033905-31343e006842/client.go:1294\ngithub.com/tikv/pd/client.(*client).ScatterRegion\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20230728033905-31343e006842/client.go:1269\ngithub.com/pingcap/tidb/br/pkg/restore/split.(*pdClient).ScatterRegion\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/client.go:510\ngithub.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).BatchSplitRegions.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/localhelper.go:403\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:57\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:75\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:56\ngithub.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).BatchSplitRegions\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/localhelper.go:390\ngithub.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).SplitAndScatterRegionByRanges.func3\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/localhelper.go:276\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650](http://ngithub.com/tikv/pd/client.(*client).scatterRegionsWithGroup/n/t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20230728033905-31343e006842/client.go:1294/ngithub.com/tikv/pd/client.(*client).ScatterRegion/n/t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20230728033905-31343e006842/client.go:1269/ngithub.com/pingcap/tidb/br/pkg/restore/split.(*pdClient).ScatterRegion/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/client.go:510/ngithub.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).BatchSplitRegions.func2/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/localhelper.go:403/ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry.func1/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:57/ngithub.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:75/ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:56/ngithub.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).BatchSplitRegions/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/localhelper.go:390/ngithub.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).SplitAndScatterRegionByRanges.func3/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/localhelper.go:276/ngolang.org/x/sync/errgroup.(*Group).Go.func1/n/t/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75/nruntime.goexit/n/t/usr/local/go/src/runtime/asm_amd64.s:1650)"]
[2023/09/08 07:27:00.631 +00:00] [INFO] [localhelper.go:300] ["batch split region"] [region_id=3426] [keys=4] [firstKey="dIAAAAAAAAHrX3KAAAAAAAAAAQ=="] [end="dIAAAAAAAAHrX3KAAAAAAAUJEQ=="]

What version of PD are you using (pd-server -V)?

[2023/09/08 05:03:01.423 +00:00] [INFO] [versioninfo.go:92] [PD] [git-hash=74ead5cbdf17df955ad7075810be3a724033ca0f]
[2023/09/08 05:03:01.423 +00:00] [INFO] [versioninfo.go:93] [PD] [git-branch=heads/refs/tags/v7.4.0-alpha]
[2023/09/08 05:03:01.423 +00:00] [INFO] [versioninfo.go:94] [PD] [utc-build-time="2023-09-07 11:36:15"]

[seiya-annie]

pd (2).log