TSAN warning signal-unsafe call inside of a signal inside in Proxy::SyncRequest

[rao-vasireddy]

WARNING: ThreadSanitizer: signal-unsafe call inside of a signal (pid=7198)

[hectorgcr]

ts3|pid7198|:10154 WARNING: ThreadSanitizer: signal-unsafe call inside of a signal (pid=7198)
ts3|pid7198|:10154     #0 operator new(unsigned long) /n/jenkins/thirdparty/yugabyte-thirdparty-2019-02-28T08_28_04/thirdparty/build/common/llvm-7.0.1.src/../../../src/llvm-7.0.1.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:65 (yb-tserver+0x4da8d7)
ts3|pid7198|:10154     #1 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__init(char const*, unsigned long) <null> (libc++.so.1+0xc7219)
ts3|pid7198|:10154     #2 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(char const*, unsigned long) /n/jenkins/thirdparty/yugabyte-thirdparty-2019-02-28T08_28_04/thirdparty/installed/tsan/libcxx/include/c++/v1/string:1799:5 (libyb_util.so+0x2821f9)
ts3|pid7198|:10154     #3 yb::(anonymous namespace)::CustomGlogFailureWriter(char const*, int) /net/centos-gcp-cloud-jenkins-slave-254/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/util/logging.cc:188 (libyb_util.so+0x2821f9)
ts3|pid7198|:10154     #4 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) <null> (libglog.so.0+0x25de9)
ts3|pid7198|:10154     #5 CallUserSignalHandler /n/jenkins/thirdparty/yugabyte-thirdparty-2019-02-28T08_28_04/thirdparty/build/common/llvm-7.0.1.src/../../../src/llvm-7.0.1.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1871 (yb-tserver+0x466481)
ts3|pid7198|:10154     #6 __tsan::ProcessPendingSignals(__tsan::ThreadState*) /n/jenkins/thirdparty/yugabyte-thirdparty-2019-02-28T08_28_04/thirdparty/build/common/llvm-7.0.1.src/../../../src/llvm-7.0.1.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1919 (yb-tserver+0x466481)
ts3|pid7198|:10154     #7 _ULx86_64_dwarf_find_save_locs <null> (libunwind.so.8+0x72a3)
ts3|pid7198|:10154     #8 yb::StackTrace::Collect(int) /net/centos-gcp-cloud-jenkins-slave-254/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/util/debug-util.cc:607:17 (libyb_util.so+0x21cd86)
ts3|pid7198|:10154     #9 yb::Mutex::CheckUnheldAndMark() /net/centos-gcp-cloud-jenkins-slave-254/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/util/mutex.cc:118:17 (libyb_util.so+0x2aa9bf)
ts3|pid7198|:10154     #10 yb::ConditionVariable::Wait() const /net/centos-gcp-cloud-jenkins-slave-254/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/util/condition_variable.cc:87:15 (libyb_util.so+0x1cd8a0)
ts3|pid7198|:10154     #11 yb::CountDownLatch::Wait() const /net/centos-gcp-cloud-jenkins-slave-254/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/util/countdown_latch.cc:47:11 (libyb_util.so+0x1ce1e5)
ts3|pid7198|:10154     #12 yb::rpc::Proxy::SyncRequest(yb::rpc::RemoteMethod const*, google::protobuf::Message const&, google::protobuf::Message*, yb::rpc::RpcController*) /net/centos-gcp-cloud-jenkins-slave-254/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/proxy.cc:266:9 (libyrpc.so+0x17c35f)
ts3|pid7198|:10154     #13 yb::master::MasterServiceProxy::TSHeartbeat(yb::master::TSHeartbeatRequestPB const&, yb::master::TSHeartbeatResponsePB*, yb::rpc::RpcController*) /net/centos-gcp-cloud-jenkins-slave-254/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/src/yb/master/master.proxy.cc:25:18 (libmaster_proto.so+0x217d96)
ts3|pid7198|:10154     #14 yb::tserver::Heartbeater::Thread::TryHeartbeat() /net/centos-gcp-cloud-jenkins-slave-254/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/tserver/heartbeater.cc:453:5 (libtserver.so+0xf0a02)
ts3|pid7198|:10154     #15 yb::tserver::Heartbeater::Thread::DoHeartbeat() /net/centos-gcp-cloud-jenkins-slave-254/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/tserver/heartbeater.cc:529:19 (libtserver.so+0xf1487)
ts3|pid7198|:10154     #16 yb::tserver::Heartbeater::Thread::RunThread() /net/centos-gcp-cloud-jenkins-slave-254/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/tserver/heartbeater.cc:576:16 (libtserver.so+0xf1944)
ts3|pid7198|:10154     #17 decltype(*(std::__1::forward<yb::tserver::Heartbeater::Thread*&>(fp0)).*fp()) std::__1::__invoke<void (yb::tserver::Heartbeater::Thread::*&)(), yb::tserver::Heartbeater::Thread*&, void>(void (yb::tserver::Heartbeater::Thread::*&)(), yb::tserver::Heartbeater::Thread*&) /n/jenkins/thirdparty/yugabyte-thirdparty-2019-02-28T08_28_04/thirdparty/installed/tsan/libcxx/include/c++/v1/type_traits:4286:1 (libtserver.so+0xfe7a3)
ts3|pid7198|:10154     #18 std::__1::__bind_return<void (yb::tserver::Heartbeater::Thread::*)(), std::__1::tuple<yb::tserver::Heartbeater::Thread*>, std::__1::tuple<>, __is_valid_bind_return<void (yb::tserver::Heartbeater::Thread::*)(), std::__1::tuple<yb::tserver::Heartbeater::Thread*>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (yb::tserver::Heartbeater::Thread::*)(), std::__1::tuple<yb::tserver::Heartbeater::Thread*>, 0ul, std::__1::tuple<> >(void (yb::tserver::Heartbeater::Thread::*&)(), std::__1::tuple<yb::tserver::Heartbeater::Thread*>&, std::__1::__tuple_indices<0ul>, std::__1::tuple<>&&) /n/jenkins/thirdparty/yugabyte-thirdparty-2019-02-28T08_28_04/thirdparty/installed/tsan/libcxx/include/c++/v1/functional:2216 (libtserver.so+0xfe7a3)
ts3|pid7198|:10154     #19 std::__1::__bind_return<void (yb::tserver::Heartbeater::Thread::*)(), std::__1::tuple<yb::tserver::Heartbeater::Thread*>, std::__1::tuple<>, __is_valid_bind_return<void (yb::tserver::Heartbeater::Thread::*)(), std::__1::tuple<yb::tserver::Heartbeater::Thread*>, std::__1::tuple<> >::value>::type std::__1::__bind<void (yb::tserver::Heartbeater::Thread::* const&)(), yb::tserver::Heartbeater::Thread* const&>::operator()<>() /n/jenkins/thirdparty/yugabyte-thirdparty-2019-02-28T08_28_04/thirdparty/installed/tsan/libcxx/include/c++/v1/functional:2249 (libtserver.so+0xfe7a3)
ts3|pid7198|:10154     #20 decltype(std::__1::forward<std::__1::__bind<void (yb::tserver::Heartbeater::Thread::* const&)(), yb::tserver::Heartbeater::Thread* const&>&>(fp)()) std::__1::__invoke<std::__1::__bind<void (yb::tserver::Heartbeater::Thread::* const&)(), yb::tserver::Heartbeater::Thread* const&>&>(std::__1::__bind<void (yb::tserver::Heartbeater::Thread::* const&)(), yb::tserver::Heartbeater::Thread* const&>&) /n/jenkins/thirdparty/yugabyte-thirdparty-2019-02-28T08_28_04/thirdparty/installed/tsan/libcxx/include/c++/v1/type_traits:4345 (libtserver.so+0xfe7a3)
ts3|pid7198|:10154     #21 void std::__1::__invoke_void_return_wrapper<void>::__call<std::__1::__bind<void (yb::tserver::Heartbeater::Thread::* const&)(), yb::tserver::Heartbeater::Thread* const&>&>(std::__1::__bind<void (yb::tserver::Heartbeater::Thread::* const&)(), yb::tserver::Heartbeater::Thread* const&>&) /n/jenkins/thirdparty/yugabyte-thirdparty-2019-02-28T08_28_04/thirdparty/installed/tsan/libcxx/include/c++/v1/__functional_base:349 (libtserver.so+0xfe7a3)
ts3|pid7198|:10154     #22 std::__1::__function::__func<std::__1::__bind<void (yb::tserver::Heartbeater::Thread::* const&)(), yb::tserver::Heartbeater::Thread* const&>, std::__1::allocator<std::__1::__bind<void (yb::tserver::Heartbeater::Thread::* const&)(), yb::tserver::Heartbeater::Thread* const&> >, void ()>::operator()() /n/jenkins/thirdparty/yugabyte-thirdparty-2019-02-28T08_28_04/thirdparty/installed/tsan/libcxx/include/c++/v1/functional:1562:12 (libtserver.so+0xfe6ad)
ts3|pid7198|:10154     #23 std::__1::function<void ()>::operator()() const /n/jenkins/thirdparty/yugabyte-thirdparty-2019-02-28T08_28_04/thirdparty/installed/tsan/libcxx/include/c++/v1/functional:1913:12 (libyb-redis.so+0xe7974)
ts3|pid7198|:10154     #24 yb::Thread::SuperviseThread(void*) /net/centos-gcp-cloud-jenkins-slave-254/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/util/thread.cc:660:3 (libyb_util.so+0x311b74)
ts3|pid7198|:10154 
ts3|pid7198|:10154 SUMMARY: ThreadSanitizer: signal-unsafe call inside of a signal /n/jenkins/thirdparty/yugabyte-thirdparty-2019-02-28T08_28_04/thirdparty/build/common/llvm-7.0.1.src/../../../src/llvm-7.0.1.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:65 in operator new(unsigned long)```

[bbaddepudi]

Looks like the test was shutting down and hit a rare race condition.

Havent seen this in the past 20+ master runs and many more user runs. Closing for now.