indexHashJoin hang in handleTask

[yibin87]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

a) use test; create table t1 ( a int, b int, index idx(b)); create table t2 ( c int, d int, index idx(d));
b) Run the script to insert data:
for i in {1..3000}
do
mysql -h127.0.0.1 -P4000 -uroot -D test -e "insert into t1 values($i, $i);"
mysql -h127.0.0.1 -P4000 -uroot -D test -e "insert into t2 values($i, $i);"
done
c) set @@tidb_max_chunk_size=32;
set @@tidb_index_join_batch_size=32;
INSERT INTO mysql.opt_rule_blacklist VALUES("topn_push_down");
ADMIN reload opt_rule_blacklist;
d)
execute the following sql statement twice:

select /*+ inl_hash_join(t2) */ t1.b from t1 left join t2  on t1.b=t2.d order by t1.b limit 1000,1;

2. What did you expect to see? (Required)

The two select statements executed successfully, and output:
+------+
| b |
+------+
| 1001 |
+------+
+------+
| b |
+------+
| 1001 |
+------+

3. What did you see instead (Required)

The first one outputs correctly, while the second one hangs.

4. What is your TiDB version? (Required)

| Release Version: v6.2.0-alpha
Edition: Community
Git Commit Hash: e0527ba
Git Branch: heads/refs/tags/v6.2.0-alpha
UTC Build Time: 2022-06-21 14:26:36
GoVersion: go1.18.2
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false
Store: tikv |

[yibin87]

From oncall issue，version v5.4.1

[guo-shaoge]

goroutine stack:

goroutine 2325 [semacquire]:
sync.runtime_Semacquire(0xc0014895d8)
    /usr/local/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc0014895d0)
    /usr/local/go/src/sync/waitgroup.go:130 +0x65
github.com/pingcap/tidb/executor.(*IndexNestedLoopHashJoin).wait4JoinWorkers(0xc000187600)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_lookup_hash_join.go:222 +0x2f
created by github.com/pingcap/tidb/executor.(*IndexNestedLoopHashJoin).startWorkers
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_lookup_hash_join.go:201 +0x610

goroutine 1465 [chan receive]:
github.com/pingcap/tidb/executor.(*IndexNestedLoopHashJoin).Close(0xc000187600, 0x4022000000000000, 0x44)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_lookup_hash_join.go:317 +0xdc
github.com/pingcap/tidb/executor.(*baseExecutor).Close(0xc000fd3340, 0xc001b76d38, 0x146af2f)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:181 +0x7e
github.com/pingcap/tidb/executor.(*LimitExec).Close(0xc000fd3340, 0x4000, 0x44)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:1151 +0x45
github.com/pingcap/tidb/executor.(*recordSet).Close(0xc001b14cd0, 0x34acb85, 0xc001b76ed0)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:171 +0x38
github.com/pingcap/tidb/session.(*execStmtResult).Close(0xc0021d2cc0, 0x34b787a, 0xc000d9ec68)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1741 +0x42
github.com/pingcap/tidb/server.(*tidbResultSet).Close(0xc001b14d20, 0x44, 0x0)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/driver_tidb.go:324 +0x4a
github.com/pingcap/tidb/parser/terror.Call(0xc001b77010)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:298 +0x3f
github.com/pingcap/tidb/server.(*clientConn).handleStmt(0xc0017b7540, 0x42c3c48, 0xc000f28150, 0x42e25f8, 0xc000dc4870, 0x618f3a0, 0x0, 0x0, 0x1, 0x0, ...)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1992 +0x494
github.com/pingcap/tidb/server.(*clientConn).handleQuery(0xc0017b7540, 0x42c3c48, 0xc00228a100, 0xc0018ea151, 0x62, 0x0, 0x0)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1819 +0x498
github.com/pingcap/tidb/server.(*clientConn).dispatch(0xc0017b7540, 0x42c3c48, 0xc00228a100, 0xc0018ea150, 0x63, 0x62, 0x0, 0x0)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1323 +0xafd
github.com/pingcap/tidb/server.(*clientConn).Run(0xc0017b7540, 0x42c3cf0, 0xc001f7b7d0)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1079 +0x29b
github.com/pingcap/tidb/server.(*Server).onConn(0xc0011aa410, 0xc0017b7540)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:553 +0xaa5
created by github.com/pingcap/tidb/server.(*Server).startNetworkListener
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:455 +0x91c

goroutine 2322 [chan send]:
github.com/pingcap/tidb/executor.(*indexHashJoinInnerWorker).handleTask.func1(0xc0005c3800, 0xc001daae70, 0xc00178be90, 0xc001898de0, 0xc0018227e0)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_lookup_hash_join.go:630 +0x98
github.com/pingcap/tidb/executor.(*indexHashJoinInnerWorker).handleTask(0xc0005c3800, 0x42c3c48, 0xc00228b300, 0xc001daae70, 0xc001898de0, 0x42de618, 0xc001df2b80, 0xc0018227e0, 0x4270ec0, 0xc00121e588)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_lookup_hash_join.go:676 +0x4a8
github.com/pingcap/tidb/executor.(*indexHashJoinInnerWorker).run(0xc0005c3800, 0x42c3c48, 0xc00228b300, 0xc0020b32a0)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_lookup_hash_join.go:506 +0x2ae
github.com/pingcap/tidb/executor.(*IndexNestedLoopHashJoin).startWorkers.func2()
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_lookup_hash_join.go:199 +0x87
github.com/pingcap/tidb/util.WithRecovery(0xc00228b3c0, 0xc0020b32e0)
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/misc.go:100 +0x4f
created by github.com/pingcap/tidb/executor.(*IndexNestedLoopHashJoin).startWorkers
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_lookup_hash_join.go:199 +0x4c5