Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

data race insertRegionToCache #19117

Closed
SunRunAway opened this issue Aug 11, 2020 · 4 comments
Closed

data race insertRegionToCache #19117

SunRunAway opened this issue Aug 11, 2020 · 4 comments
Labels
component/test severity/moderate sig/transaction SIG:Transaction type/bug The issue is confirmed as a bug.

Comments

@SunRunAway
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

[2020-08-11T03:15:46.166Z] ==================

[2020-08-11T03:15:46.166Z] WARNING: DATA RACE

[2020-08-11T03:15:46.166Z] Read at 0x00c0c7e3e4d8 by goroutine 63:

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/store/tikv.(*RegionCache).insertRegionToCache()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:823 +0x42f

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/store/tikv.(*RegionCache).findRegionByKey()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:532 +0x829

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/store/tikv.(*RegionCache).LocateKey()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:495 +0x89

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/store/tikv.(*tikvSnapshot).get()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/snapshot.go:389 +0x6f1

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/store/tikv.(*tikvSnapshot).Get()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/snapshot.go:331 +0x3ea

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/kv.(*unionStore).Get()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/kv/union_store.go:98 +0x2de

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/store/tikv.(*tikvTxn).Get()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/txn.go:138 +0xb1

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/structure.(*TxStructure).loadListMeta()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/structure/list.go:217 +0xcd

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/structure.(*TxStructure).LIndex()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/structure/list.go:163 +0xa6

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/meta.(*Meta).getDDLJob()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/meta/meta.go:603 +0x9a

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/meta.(*Meta).GetDDLJobByIdx()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/meta/meta.go:632 +0xf4

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/ddl.(*worker).getFirstDDLJob()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:279 +0x49

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/ddl.(*worker).handleDDLJobQueue.func1()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:442 +0x139

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/kv.RunInNewTxn()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/kv/txn.go:47 +0x110

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/ddl.(*worker).handleDDLJobQueue()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:433 +0x1b6

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/ddl.(*worker).start()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:154 +0x478

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/ddl.(*worker).handleDDLJobQueue()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:433 +0x1b6

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/ddl.(*worker).start()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:154 +0x478

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/ddl.(*worker).handleDDLJobQueue()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:433 +0x1b6

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/ddl.(*worker).start()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:154 +0x478

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/ddl.(*worker).handleDDLJobQueue()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:433 +0x1b6

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/ddl.(*worker).start()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:154 +0x478

[2020-08-11T03:15:46.166Z] 

[2020-08-11T03:15:46.166Z] Previous write at 0x00c0c7e3e4d8 by goroutine 841:

[2020-08-11T03:15:46.166Z]   sync/atomic.StoreInt32()

[2020-08-11T03:15:46.166Z]       /usr/local/go/src/runtime/race_amd64.s:229 +0xb

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/store/tikv.(*RegionCache).GetTiFlashRPCContext()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:454 +0x3a1

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/store/tikv.buildBatchCopTasks()

[2020-08-11T03:15:46.166Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/batch_coprocessor.go:125 +0x1a5

[2020-08-11T03:15:46.166Z]   github.com/pingcap/tidb/store/tikv.(*batchCopIterator).retryBatchCopTask()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/batch_coprocessor.go:324 +0x1e5

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/store/tikv.(*batchCopIterator).handleTaskOnce()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/batch_coprocessor.go:364 +0x101d

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/store/tikv.(*batchCopIterator).handleTask()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/batch_coprocessor.go:305 +0x1a1

[2020-08-11T03:15:46.167Z] 

[2020-08-11T03:15:46.167Z] Goroutine 63 (running) created at:

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/ddl.(*ddl).Start()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl.go:340 +0xb19

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/domain.(*Domain).Init()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/domain/domain.go:724 +0x4c8

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.(*domainMap).Get.func1()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/tidb.go:77 +0x5bd

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/util.RunWithRetry()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/misc.go:61 +0x90

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.(*domainMap).Get()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/tidb.go:69 +0x269

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.createSessionWithOpt()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:1860 +0x7c

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.createSession()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:1856 +0x4f

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.BootstrapSession()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:1745 +0xfe

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1181 +0x3d9

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1179 +0x3ac

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1177 +0x37f

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1175 +0x352

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1173 +0x325

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1171 +0x2f8

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1169 +0x2cb

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1167 +0x29e

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1165 +0x271

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1163 +0x244

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1161 +0x217

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1159 +0x1ea

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1157 +0x1bd

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1156 +0x190

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1155 +0x163

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1154 +0x136

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1152 +0x109

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1150 +0xdc

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.doDDLWorks()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:1148 +0x5b

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.bootstrap()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/bootstrap.go:310 +0x1a1

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.runInBootstrapSession()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:1846 +0xe2

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/session.BootstrapSession()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:1740 +0xb69

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/store/tikv_test.newStoreWithBootstrap()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/batch_coprocessor_test.go:67 +0x167

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/store/tikv_test.(*testBatchCopSuite).TestStoreSwitchPeer()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/batch_coprocessor_test.go:122 +0x4f

[2020-08-11T03:15:46.167Z]   runtime.call32()

[2020-08-11T03:15:46.167Z]       /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a

[2020-08-11T03:15:46.167Z]   reflect.Value.Call()

[2020-08-11T03:15:46.167Z]       /usr/local/go/src/reflect/value.go:321 +0xd3

[2020-08-11T03:15:46.167Z]   github.com/pingcap/check.(*suiteRunner).forkTest.func1()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:850 +0x9aa

[2020-08-11T03:15:46.167Z]   github.com/pingcap/check.(*suiteRunner).forkCall.func1()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:739 +0x113

[2020-08-11T03:15:46.167Z] 

[2020-08-11T03:15:46.167Z] Goroutine 841 (running) created at:

[2020-08-11T03:15:46.167Z]   github.com/pingcap/tidb/store/tikv.(*batchCopIterator).run()

[2020-08-11T03:15:46.167Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/batch_coprocessor.go:232 +0xc5

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/tidb_ghpr_unit_test/detail/tidb_ghpr_unit_test/45891/pipeline
@SunRunAway SunRunAway added type/bug The issue is confirmed as a bug. component/test labels Aug 11, 2020
@SunRunAway
Copy link
Contributor Author

Write:

tidb/store/tikv/region_cache.go

Line 454 in afb4875

atomic.StoreInt32(&regionStore.workTiFlashIdx, int32(accessIdx))

Read:

tidb/store/tikv/region_cache.go

Line 823 in afb4875

cachedRegion.getStore().workTiFlashIdx = old.(*btreeItem).cachedRegion.getStore().workTiFlashIdx

@SunRunAway
Copy link
Contributor Author

@lzmhhh123 PTAL

@jebter jebter added the sig/transaction SIG:Transaction label Dec 7, 2021
@sticnarf
Copy link
Contributor

Closed by #19198

@github-actions
Copy link

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/test severity/moderate sig/transaction SIG:Transaction type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@SunRunAway @jebter @sticnarf @sre-bot