Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data race when updating store progress #8036

Closed
rleungx opened this issue Apr 8, 2024 · 0 comments · Fixed by #8037
Closed

Data race when updating store progress #8036

rleungx opened this issue Apr 8, 2024 · 0 comments · Fixed by #8037
Assignees
@CabinfeverB
Labels
severity/minor type/bug The issue is confirmed as a bug.

Comments

@rleungx
Copy link
Member

rleungx commented Apr 8, 2024

Bug Report

What did you do?

run test

What did you expect to see?

ci pass

What did you see instead?

2024-04-08T04:09:21.4538217Z ==================
2024-04-08T04:09:21.4538320Z WARNING: DATA RACE
2024-04-08T04:09:21.4538472Z Write at 0x00c0011df288 by goroutine 1513769:
2024-04-08T04:09:21.4538854Z   github.com/tikv/pd/server/cluster.(*schedulingController).initCoordinatorLocked()
2024-04-08T04:09:21.4539349Z       /home/runner/work/pd/pd/server/cluster/scheduling_controller.go:114 +0x186
2024-04-08T04:09:21.4539713Z   github.com/tikv/pd/server/cluster.(*schedulingController).startSchedulingJobs()
2024-04-08T04:09:21.4540188Z       /home/runner/work/pd/pd/server/cluster/scheduling_controller.go:96 +0x205
2024-04-08T04:09:21.4540458Z   github.com/tikv/pd/server/cluster.(*RaftCluster).checkServices()
2024-04-08T04:09:21.4540760Z       /home/runner/work/pd/pd/server/cluster/cluster.go:357 +0x2d5
2024-04-08T04:09:21.4541057Z   github.com/tikv/pd/server/cluster.(*RaftCluster).runServiceCheckJob()
2024-04-08T04:09:21.4541363Z       /home/runner/work/pd/pd/server/cluster/cluster.go:390 +0x2f1
2024-04-08T04:09:21.4541615Z   github.com/tikv/pd/server/cluster.(*RaftCluster).Start.func2()
2024-04-08T04:09:21.4541920Z       /home/runner/work/pd/pd/server/cluster/cluster.go:339 +0x33
2024-04-08T04:09:21.4541930Z 
2024-04-08T04:09:21.4542110Z Previous read at 0x00c0011df288 by goroutine 1513771:
2024-04-08T04:09:21.4542388Z   github.com/tikv/pd/server/cluster.(*RaftCluster).updateProgress()
2024-04-08T04:09:21.4542705Z       /home/runner/work/pd/pd/server/cluster/cluster.go:1943 +0x24f
2024-04-08T04:09:21.4542976Z   github.com/tikv/pd/server/cluster.(*RaftCluster).checkStores()
2024-04-08T04:09:21.4543295Z       /home/runner/work/pd/pd/server/cluster/cluster.go:1753 +0x2071
2024-04-08T04:09:21.4543600Z   github.com/tikv/pd/server/cluster.(*RaftCluster).runNodeStateCheckJob()
2024-04-08T04:09:21.4543911Z       /home/runner/work/pd/pd/server/cluster/cluster.go:693 +0x2f1
2024-04-08T04:09:21.4544164Z   github.com/tikv/pd/server/cluster.(*RaftCluster).Start.func4()
2024-04-08T04:09:21.4544470Z       /home/runner/work/pd/pd/server/cluster/cluster.go:341 +0x33
2024-04-08T04:09:21.4544477Z 
2024-04-08T04:09:21.4544607Z Goroutine 1513769 (running) created at:
2024-04-08T04:09:21.4544827Z   github.com/tikv/pd/server/cluster.(*RaftCluster).Start()
2024-04-08T04:09:21.4545129Z       /home/runner/work/pd/pd/server/cluster/cluster.go:339 +0xdf3
2024-04-08T04:09:21.4545343Z   github.com/tikv/pd/server.(*Server).bootstrapCluster()
2024-04-08T04:09:21.4545599Z       /home/runner/work/pd/pd/server/server.go:784 +0x1ccf
2024-04-08T04:09:21.4545793Z   github.com/tikv/pd/server.(*GrpcServer).Bootstrap()
2024-04-08T04:09:21.4546082Z       /home/runner/work/pd/pd/server/grpc_service.go:629 +0x989
2024-04-08T04:09:21.4546304Z   github.com/tikv/pd/tests.(*TestServer).BootstrapCluster()
2024-04-08T04:09:21.4546558Z       /home/runner/work/pd/pd/tests/cluster.go:417 +0x62e
2024-04-08T04:09:21.4546848Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).startCluster()
2024-04-08T04:09:21.4547171Z       /home/runner/work/pd/pd/tests/testutil.go:354 +0x9af
2024-04-08T04:09:21.4547487Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).RunTestInAPIMode()
2024-04-08T04:09:21.4547735Z       /home/runner/work/pd/pd/tests/testutil.go:308 +0x2c4
2024-04-08T04:09:21.4547900Z   github.com/pingcap/failpoint.parseTerm()
2024-04-08T04:09:21.4548631Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/terms.go:149 +0x364
2024-04-08T04:09:21.4548785Z   github.com/pingcap/failpoint.parse()
2024-04-08T04:09:21.4549436Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/terms.go:126 +0xa5
2024-04-08T04:09:21.4549589Z   github.com/pingcap/failpoint.newTerms()
2024-04-08T04:09:21.4550235Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/terms.go:98 +0x3e
2024-04-08T04:09:21.4550428Z   github.com/pingcap/failpoint.(*Failpoint).Enable()
2024-04-08T04:09:21.4551103Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/failpoint.go:54 +0x3e
2024-04-08T04:09:21.4551295Z   github.com/pingcap/failpoint.(*Failpoints).Enable()
2024-04-08T04:09:21.4551983Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/failpoints.go:105 +0x296
2024-04-08T04:09:21.4552126Z   github.com/pingcap/failpoint.Enable()
2024-04-08T04:09:21.4552874Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/failpoints.go:222 +0x134
2024-04-08T04:09:21.4553242Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).RunTestInAPIMode()
2024-04-08T04:09:21.4553498Z       /home/runner/work/pd/pd/tests/testutil.go:301 +0x135
2024-04-08T04:09:21.4553660Z   github.com/pingcap/failpoint.parseTerm()
2024-04-08T04:09:21.4554310Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/terms.go:149 +0x364
2024-04-08T04:09:21.4554461Z   github.com/pingcap/failpoint.parse()
2024-04-08T04:09:21.4555101Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/terms.go:126 +0xa5
2024-04-08T04:09:21.4555258Z   github.com/pingcap/failpoint.newTerms()
2024-04-08T04:09:21.4555889Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/terms.go:98 +0x3e
2024-04-08T04:09:21.4556080Z   github.com/pingcap/failpoint.(*Failpoint).Enable()
2024-04-08T04:09:21.4556748Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/failpoint.go:54 +0x3e
2024-04-08T04:09:21.4556943Z   github.com/pingcap/failpoint.(*Failpoints).Enable()
2024-04-08T04:09:21.4557625Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/failpoints.go:105 +0x296
2024-04-08T04:09:21.4557767Z   github.com/pingcap/failpoint.Enable()
2024-04-08T04:09:21.4558446Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/failpoints.go:222 +0xf6
2024-04-08T04:09:21.4558757Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).RunTestInAPIMode()
2024-04-08T04:09:21.4559012Z       /home/runner/work/pd/pd/tests/testutil.go:300 +0xf7
2024-04-08T04:09:21.4559425Z   github.com/tikv/pd/tests/integrations/mcs/scheduling_test.(*apiTestSuite).TestAPIForward()
2024-04-08T04:09:21.4559868Z       /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/api_test.go:109 +0x6f
2024-04-08T04:09:21.4559985Z   runtime.call16()
2024-04-08T04:09:21.4560305Z       /opt/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:747 +0x42
2024-04-08T04:09:21.4560427Z   reflect.Value.Call()
2024-04-08T04:09:21.4560750Z       /opt/hostedtoolcache/go/1.21.8/x64/src/reflect/value.go:380 +0xb5
2024-04-08T04:09:21.4560930Z   github.com/stretchr/testify/suite.Run.func1()
2024-04-08T04:09:21.4561405Z       /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x766
2024-04-08T04:09:21.4561579Z   testing.tRunner()
2024-04-08T04:09:21.4561935Z       /opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1595 +0x261
2024-04-08T04:09:21.4562051Z   testing.(*T).Run.func1()
2024-04-08T04:09:21.4562381Z       /opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1648 +0x44
2024-04-08T04:09:21.4562388Z 
2024-04-08T04:09:21.4562512Z Goroutine 1513771 (running) created at:
2024-04-08T04:09:21.4562736Z   github.com/tikv/pd/server/cluster.(*RaftCluster).Start()
2024-04-08T04:09:21.4563102Z       /home/runner/work/pd/pd/server/cluster/cluster.go:341 +0xef3
2024-04-08T04:09:21.4563319Z   github.com/tikv/pd/server.(*Server).bootstrapCluster()
2024-04-08T04:09:21.4563577Z       /home/runner/work/pd/pd/server/server.go:784 +0x1ccf
2024-04-08T04:09:21.4563769Z   github.com/tikv/pd/server.(*GrpcServer).Bootstrap()
2024-04-08T04:09:21.4564062Z       /home/runner/work/pd/pd/server/grpc_service.go:629 +0x989
2024-04-08T04:09:21.4564284Z   github.com/tikv/pd/tests.(*TestServer).BootstrapCluster()
2024-04-08T04:09:21.4564545Z       /home/runner/work/pd/pd/tests/cluster.go:417 +0x62e
2024-04-08T04:09:21.4564834Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).startCluster()
2024-04-08T04:09:21.4565080Z       /home/runner/work/pd/pd/tests/testutil.go:354 +0x9af
2024-04-08T04:09:21.4565388Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).RunTestInAPIMode()
2024-04-08T04:09:21.4565633Z       /home/runner/work/pd/pd/tests/testutil.go:308 +0x2c4
2024-04-08T04:09:21.4565850Z   github.com/pingcap/failpoint.parseTerm()
2024-04-08T04:09:21.4566687Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/terms.go:149 +0x364
2024-04-08T04:09:21.4566937Z   github.com/pingcap/failpoint.parse()
2024-04-08T04:09:21.4567607Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/terms.go:126 +0xa5
2024-04-08T04:09:21.4567768Z   github.com/pingcap/failpoint.newTerms()
2024-04-08T04:09:21.4568404Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/terms.go:98 +0x3e
2024-04-08T04:09:21.4568600Z   github.com/pingcap/failpoint.(*Failpoint).Enable()
2024-04-08T04:09:21.4569282Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/failpoint.go:54 +0x3e
2024-04-08T04:09:21.4569477Z   github.com/pingcap/failpoint.(*Failpoints).Enable()
2024-04-08T04:09:21.4570167Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/failpoints.go:105 +0x296
2024-04-08T04:09:21.4570311Z   github.com/pingcap/failpoint.Enable()
2024-04-08T04:09:21.4570995Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/failpoints.go:222 +0x134
2024-04-08T04:09:21.4571305Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).RunTestInAPIMode()
2024-04-08T04:09:21.4571558Z       /home/runner/work/pd/pd/tests/testutil.go:301 +0x135
2024-04-08T04:09:21.4571723Z   github.com/pingcap/failpoint.parseTerm()
2024-04-08T04:09:21.4572370Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/terms.go:149 +0x364
2024-04-08T04:09:21.4572517Z   github.com/pingcap/failpoint.parse()
2024-04-08T04:09:21.4573153Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/terms.go:126 +0xa5
2024-04-08T04:09:21.4573313Z   github.com/pingcap/failpoint.newTerms()
2024-04-08T04:09:21.4573945Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/terms.go:98 +0x3e
2024-04-08T04:09:21.4574135Z   github.com/pingcap/failpoint.(*Failpoint).Enable()
2024-04-08T04:09:21.4574803Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/failpoint.go:54 +0x3e
2024-04-08T04:09:21.4574996Z   github.com/pingcap/failpoint.(*Failpoints).Enable()
2024-04-08T04:09:21.4575678Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/failpoints.go:105 +0x296
2024-04-08T04:09:21.4575904Z   github.com/pingcap/failpoint.Enable()
2024-04-08T04:09:21.4576585Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20220801062533-2eaa32854a6c/failpoints.go:222 +0xf6
2024-04-08T04:09:21.4576891Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).RunTestInAPIMode()
2024-04-08T04:09:21.4577147Z       /home/runner/work/pd/pd/tests/testutil.go:300 +0xf7
2024-04-08T04:09:21.4577565Z   github.com/tikv/pd/tests/integrations/mcs/scheduling_test.(*apiTestSuite).TestAPIForward()
2024-04-08T04:09:21.4578091Z       /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/api_test.go:109 +0x6f
2024-04-08T04:09:21.4578208Z   runtime.call16()
2024-04-08T04:09:21.4578529Z       /opt/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:747 +0x42
2024-04-08T04:09:21.4578649Z   reflect.Value.Call()
2024-04-08T04:09:21.4578973Z       /opt/hostedtoolcache/go/1.21.8/x64/src/reflect/value.go:380 +0xb5
2024-04-08T04:09:21.4579152Z   github.com/stretchr/testify/suite.Run.func1()
2024-04-08T04:09:21.4579593Z       /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x766
2024-04-08T04:09:21.4579698Z   testing.tRunner()
2024-04-08T04:09:21.4580052Z       /opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1595 +0x261
2024-04-08T04:09:21.4580165Z   testing.(*T).Run.func1()
2024-04-08T04:09:21.4580498Z       /opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1648 +0x44
2024-04-08T04:09:21.4580585Z ==================

Introduced by #7722

What version of PD are you using (pd-server -V)?

master
@rleungx rleungx added the type/bug The issue is confirmed as a bug. label Apr 8, 2024
@CabinfeverB CabinfeverB mentioned this issue Apr 8, 2024
Merged
@ti-chi-bot ti-chi-bot bot closed this as completed in #8037 Apr 9, 2024
ti-chi-bot bot added a commit that referenced this issue Apr 9, 2024
@CabinfeverB @ti-chi-bot
cluster: fix race when getting coordinator (#8037)
a1e957d 
close #8036

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CabinfeverB CabinfeverB

Labels
severity/minor type/bug The issue is confirmed as a bug.
Projects
Questions and Bug Reports
Status: Closed
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cluster: fix race when getting coordinator CabinfeverB/pd
2 participants
@CabinfeverB @rleungx