tikv: fix panic when recycle non-superbatch idle conn

[lysu]

What problem does this PR solve?

Problem Summary:

tidb-tikv will use superbatch conn as default, but "cluster table" will establish non-superbatch connection between tidb-tidb.

tidb/store/tikv/client.go

Line 322 in af0a49c

enableBatch := req.StoreTp != kv.TiDB

and superbatch due use streaming, it has recycle idle conn logic to avoid goroutine leak.

but "recycle idle conn logic" doesn't work well when conns map mix superbatch and non-superbatch conn, and will meet segv fault.

[2020/04/10 14:56:42.816 +08:00] [ERROR] [coprocessor.go:657] ["copIteratorWork meet panic"] [r="\"invalid memory address or nil pointer dereference\""] ["stack trace"="github.com/pingcap/tidb/store/tikv.(*copIteratorWorker).handleTask.func1\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:659\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:679\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:199\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:394\ngithub.com/pingcap/tidb/store/tikv.(*rpcClient).recycleDieConnArray\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:658\ngithub.com/pingcap/tidb/store/tikv.(*rpcClient).recycleDieConnArray\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:658\ngithub.com/pingcap/tidb/store/tikv.(*rpcClient).SendRequest\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/client.go:319\ngithub.com/pingcap/tidb/store/tikv.(*RegionRequestSender).sendReqToRegion\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/region_request.go:199\ngithub.com/pingcap/tidb/store/tikv.(*RegionRequestSender).SendReqCtx\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/region_request.go:162\ngithub.com/pingcap/tidb/store/tikv.(*clientHelper).SendReqCtx\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:814\ngithub.com/pingcap/tidb/store/tikv.(*copIteratorWorker).handleTaskOnce\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:729\ngithub.com/pingcap/tidb/store/tikv.(*copIteratorWorker).handleTask\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:667\ngithub.com/pingcap/tidb/store/tikv.(*copIteratorWorker).run\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:489"]

What is changed and how it works?

What's Changed:

only recycle superbatch conns

How it Works:

only recycle superbatch conns

Related changes

• Need to cherry-pick to the release branch

Check List

Tests

• Manual test (add detailed scripts or steps below)

setup a tidb cluster with multi-tidb instances, and remove a tikv store, then continue give pressure on tidb to trigger "recycle conn logic"

Side effects

• n/a

Release note

• tikv: fix panic when recycle non-superbatch idle conn between tidb instances

---

This change is 

[lysu]

/run-all-tests

[codecov]

Codecov Report

Merging #16299 into master will not change coverage by %.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #16299   +/-   ##
===========================================
  Coverage   80.4598%   80.4598%           
===========================================
  Files           506        506           
  Lines        136135     136135           
===========================================
  Hits         109534     109534           
  Misses        18106      18106           
  Partials       8495       8495           

[ngaut]

LGTM

[zz-jason]

LGTM

[sre-bot]

/run-all-tests

[sre-bot]

@lysu merge failed.

[sre-bot]

cherry pick to release-4.0 in PR #16303