ThreadSanitizer reports data race in gtest_dm_delta_merge_store.cpp.

[JinheLin]

Bug Report

ThreadSanitizer reports some data race of gtest_dm_delta_merge_store.cpp.

It seems that this data race is caused by foreground thread and background thread call flushCache concurrently.

How can we fix this problem in unit test? And does this problem occur in a normal process?

Here is some message (All gtests_dbms ThreadSanitizer result):

Thread 1:

[2022-03-22T03:31:20.999Z]   Write of size 8 at 0x7b2400009150 by main thread (mutexes: write M34272):
[2022-03-22T03:31:20.999Z]     #0 std::__1::enable_if<(is_move_constructible<std::__1::vector<std::__1::shared_ptr<DB::DM::ColumnFilePersisted>, std::__1::allocator<std::__1::shared_ptr<DB::DM::ColumnFilePersisted> > >*>::value) && (is_move_assignable<std::__1::vector<std::__1::shared_ptr<DB::DM::ColumnFilePersisted>, std::__1::allocator<std::__1::shared_ptr<DB::DM::ColumnFilePersisted> > >*>::value), void>::type std::__1::swap<std::__1::vector<std::__1::shared_ptr<DB::DM::ColumnFilePersisted>, std::__1::allocator<std::__1::shared_ptr<DB::DM::ColumnFilePersisted> > >*>(std::__1::vector<std::__1::shared_ptr<DB::DM::ColumnFilePersisted>, std::__1::allocator<std::__1::shared_ptr<DB::DM::ColumnFilePersisted> > >*&, std::__1::vector<std::__1::shared_ptr<DB::DM::ColumnFilePersisted>, std::__1::allocator<std::__1::shared_ptr<DB::DM::ColumnFilePersisted> > >*&) /usr/local/bin/../include/c++/v1/__utility/swap.h:39:7 (gtests_dbms+0xe35660f)
[2022-03-22T03:31:20.999Z]     #1 std::__1::vector<std::__1::vector<std::__1::shared_ptr<DB::DM::ColumnFilePersisted>, std::__1::allocator<std::__1::shared_ptr<DB::DM::ColumnFilePersisted> > >, std::__1::allocator<std::__1::vector<std::__1::shared_ptr<DB::DM::ColumnFilePersisted>, std::__1::allocator<std::__1::shared_ptr<DB::DM::ColumnFilePersisted> > > > >::swap(std::__1::vector<std::__1::vector<std::__1::shared_ptr<DB::DM::ColumnFilePersisted>, std::__1::allocator<std::__1::shared_ptr<DB::DM::ColumnFilePersisted> > >, std::__1::allocator<std::__1::vector<std::__1::shared_ptr<DB::DM::ColumnFilePersisted>, std::__1::allocator<std::__1::shared_ptr<DB::DM::ColumnFilePersisted> > > > >&) /usr/local/bin/../include/c++/v1/vector:2076:5 (gtests_dbms+0xe35660f)
[2022-03-22T03:31:20.999Z]     #2 DB::DM::ColumnFilePersistedSet::appendPersistedColumnFilesToLevel0(std::__1::vector<std::__1::shared_ptr<DB::DM::ColumnFilePersisted>, std::__1::allocator<std::__1::shared_ptr<DB::DM::ColumnFilePersisted> > > const&, DB::DM::WriteBatches&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Delta/ColumnFilePersistedSet.cpp:320:28 (gtests_dbms+0xe35660f)
[2022-03-22T03:31:20.999Z]     #3 DB::DM::ColumnFileFlushTask::commit(std::__1::shared_ptr<DB::DM::ColumnFilePersistedSet>&, DB::DM::WriteBatches&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Delta/ColumnFileFlushTask.cpp:99:30 (gtests_dbms+0xe34f0e1)
[2022-03-22T03:31:20.999Z]     #4 DB::DM::DeltaValueSpace::flush(DB::DM::DMContext&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Delta/DeltaValueSpace.cpp:193:26 (gtests_dbms+0xe362a36)
[2022-03-22T03:31:20.999Z]     #5 DB::DM::Segment::flushCache(DB::DM::DMContext&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Segment.cpp:1278:19 (gtests_dbms+0xe245b83)
[2022-03-22T03:31:20.999Z]     #6 DB::DM::DeltaMergeStore::flushCache(std::__1::shared_ptr<DB::DM::DMContext> const&, DB::DM::RowKeyRange const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:915:26 (gtests_dbms+0xe201417)
[2022-03-22T03:31:20.999Z]     #7 DB::DM::DeltaMergeStore::flushCache(DB::Context const&, DB::DM::RowKeyRange const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore.h:369:9 (gtests_dbms+0x6957ffc)
[2022-03-22T03:31:20.999Z]     #8 DB::DM::tests::DeltaMergeStoreRWTest_Split_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_delta_merge_store.cpp:1442:20 (gtests_dbms+0x6907c88)
[2022-03-22T03:31:20.999Z]     #9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2401:10 (gtests_dbms+0xc7cff55)
[2022-03-22T03:31:20.999Z]     #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2437:14 (gtests_dbms+0xc7cff55)
[2022-03-22T03:31:20.999Z]     #11 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2473:5 (gtests_dbms+0xc7afc75)
[2022-03-22T03:31:20.999Z]     #12 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2655:11 (gtests_dbms+0xc7b1441)
[2022-03-22T03:31:21.000Z]     #13 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2773:28 (gtests_dbms+0xc7b20fd)
[2022-03-22T03:31:21.000Z]     #14 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4673:43 (gtests_dbms+0xc7bd480)
[2022-03-22T03:31:21.000Z]     #15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2401:10 (gtests_dbms+0xc7d1015)
[2022-03-22T03:31:21.000Z]     #16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2437:14 (gtests_dbms+0xc7d1015)
[2022-03-22T03:31:21.000Z]     #17 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4281:10 (gtests_dbms+0xc7bc903)
[2022-03-22T03:31:21.000Z]     #18 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2237:46 (gtests_dbms+0x6e12e57)
[2022-03-22T03:31:21.000Z]     #19 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:36:16 (gtests_dbms+0x6e12e57)

Thread 2：

[2022-03-22T03:31:21.000Z]   Previous read of size 8 at 0x7b2400009150 by thread T9:
[2022-03-22T03:31:21.000Z]     #0 std::__1::vector<std::__1::vector<std::__1::shared_ptr<DB::DM::ColumnFilePersisted>, std::__1::allocator<std::__1::shared_ptr<DB::DM::ColumnFilePersisted> > >, std::__1::allocator<std::__1::vector<std::__1::shared_ptr<DB::DM::ColumnFilePersisted>, std::__1::allocator<std::__1::shared_ptr<DB::DM::ColumnFilePersisted> > > > >::size() const /usr/local/bin/../include/c++/v1/vector:680:61 (gtests_dbms+0xe35dd84)
[2022-03-22T03:31:21.000Z]     #1 DB::DM::ColumnFilePersistedSet::info() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Delta/ColumnFilePersistedSet.h:87:55 (gtests_dbms+0xe35dd84)
[2022-03-22T03:31:21.000Z]     #2 DB::DM::DeltaValueSpace::info() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Delta/DeltaValueSpace.h:112:81 (gtests_dbms+0xe368047)
[2022-03-22T03:31:21.000Z]     #3 DB::DM::DeltaValueSpace::flush(DB::DM::DMContext&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Delta/DeltaValueSpace.cpp:144:5 (gtests_dbms+0xe361d43)
[2022-03-22T03:31:21.000Z]     #4 DB::DM::Segment::flushCache(DB::DM::DMContext&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Segment.cpp:1278:19 (gtests_dbms+0xe245b83)
[2022-03-22T03:31:21.000Z]     #5 DB::DM::DeltaMergeStore::handleBackgroundTask(bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:1451:27 (gtests_dbms+0xe20efe2)
[2022-03-22T03:31:21.000Z]     #6 DB::DM::DeltaMergeStore::setUpBackgroundTask(std::__1::shared_ptr<DB::DM::DMContext> const&)::$_59::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:336:70 (gtests_dbms+0xe21d935)
[2022-03-22T03:31:21.000Z]     #7 decltype(static_cast<DB::DM::DeltaMergeStore::setUpBackgroundTask(std::__1::shared_ptr<DB::DM::DMContext> const&)::$_59&>(fp)()) std::__1::__invoke<DB::DM::DeltaMergeStore::setUpBackgroundTask(std::__1::shared_ptr<DB::DM::DMContext> const&)::$_59&>(DB::DM::DeltaMergeStore::setUpBackgroundTask(std::__1::shared_ptr<DB::DM::DMContext> const&)::$_59&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xe21d935)
[2022-03-22T03:31:21.000Z]     #8 bool std::__1::__invoke_void_return_wrapper<bool, false>::__call<DB::DM::DeltaMergeStore::setUpBackgroundTask(std::__1::shared_ptr<DB::DM::DMContext> const&)::$_59&>(DB::DM::DeltaMergeStore::setUpBackgroundTask(std::__1::shared_ptr<DB::DM::DMContext> const&)::$_59&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:30:16 (gtests_dbms+0xe21d935)
[2022-03-22T03:31:21.000Z]     #9 std::__1::__function::__alloc_func<DB::DM::DeltaMergeStore::setUpBackgroundTask(std::__1::shared_ptr<DB::DM::DMContext> const&)::$_59, std::__1::allocator<DB::DM::DeltaMergeStore::setUpBackgroundTask(std::__1::shared_ptr<DB::DM::DMContext> const&)::$_59>, bool ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0xe21d935)
[2022-03-22T03:31:21.000Z]     #10 std::__1::__function::__func<DB::DM::DeltaMergeStore::setUpBackgroundTask(std::__1::shared_ptr<DB::DM::DMContext> const&)::$_59, std::__1::allocator<DB::DM::DeltaMergeStore::setUpBackgroundTask(std::__1::shared_ptr<DB::DM::DMContext> const&)::$_59>, bool ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0xe21d935)
[2022-03-22T03:31:21.000Z]     #11 std::__1::__function::__value_func<bool ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xe0f2eaa)
[2022-03-22T03:31:21.000Z]     #12 std::__1::function<bool ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xe0f2eaa)
[2022-03-22T03:31:21.000Z]     #13 DB::BackgroundProcessingPool::threadFunction() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/BackgroundProcessingPool.cpp:225:33 (gtests_dbms+0xe0f2eaa)
[2022-03-22T03:31:21.000Z]     #14 DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_1::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/BackgroundProcessingPool.cpp:84:39 (gtests_dbms+0xe0f3730)
[2022-03-22T03:31:21.000Z]     #15 decltype(static_cast<DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_1>(fp)()) std::__1::__invoke<DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_1>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_1&&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xe0f3730)
[2022-03-22T03:31:21.001Z]     #16 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_1>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_1>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xe0f3730)
[2022-03-22T03:31:21.001Z]     #17 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_1> >(void*) /usr/local/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xe0f3730)

1. Minimal reproduce step (Required)

Compile gtests_dbms in ThreadSanitizer mode and run it.

2. What did you expect to see? (Required)

No data race.

3. What did you see instead (Required)

Some data race happen.

4. What is your TiFlash version? (Required)

master