The status code of Not Leader is sometimes wrong

[YuJuncen]

Bug Report

What did you do?

Try to access the ScanRegions gRPC endpoint in a non-leader PD.

What did you expect to see?

The gRPC status code should be Unavailable, according to the definition of ErrNotLeader.

What did you see instead?

The error code is Unknown, and the error message is nested.

rpc error: code = Unknown desc = rpc error: code = Unavailable desc = not leader
                                 ^ The real `ErrNotLeader`.
github.com/tikv/pd/client.(*client).ScanRegions
	/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20220307081149-841fa61e9710/client.go:1500
github.com/pingcap/tidb/br/pkg/restore.(*pdClient).ScanRegions
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split_client.go:502
github.com/pingcap/tidb/br/pkg/restore.PaginateScanRegion.func1
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split.go:452

What version of PD are you using (pd-server -V)?

Nightly

Notes

The cause of this error may be:

• We have wrapped the error at here:
  

  pd/server/grpc_service.go

  Lines 1490 to 1493 in 25bd92b

  	func (s *GrpcServer) validateRequest(header *pdpb.RequestHeader) error {
  	if s.IsClosed() || !s.member.IsLeader() {
  	return errors.WithStack(ErrNotLeader)
  	}

• However, gRPC would try to convert the error to status.Status, via the method GRPCStatus...
  https://github.com/grpc/grpc-go/blob/5682cc6a321507ce2a865beb8dd4bed9a5db0e06/status/status.go#L87-L96
  ...which would always fail for errors.withStack and wrap it with UNKNOWN status.

[mayjiang0203]

/found automation
/severity major
/assign rleungx

[rleungx]

Another problem is we also wrap the error of send or receive, which makes the status code change to unknown.

[rleungx]

There might be two ways to solve this bug: one is we abandon the stack, another is we turn it into gRPC status, modify some fields and wrap it again.

[nolouch]

Does this issue already fixed? @rleungx

[rleungx]

The issue mentioned problem is fixed.

Another problem is we also wrap the error of send or receive, which makes the status code change to unknown.

This has not been fixed yet

[rleungx]

Another problem is we also wrap the error of send or receive, which makes the status code change to unknown.

Create another issue to track the problem. See #5020