Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CameraServer] segfault during static destruction #7685

Open
mcm001 opened this issue Jan 15, 2025 · 1 comment · May be fixed by #7686
Open

[CameraServer] segfault during static destruction #7685

mcm001 opened this issue Jan 15, 2025 · 1 comment · May be fixed by #7686
Labels
type: bug Something isn't working.

Comments

@mcm001
Copy link
Contributor

mcm001 commented Jan 15, 2025

Describe the bug

See PhotonVision/photonvision#1718 - during gtest teardown, we randomly see segfaults.

**Additional context**
Running a photonlibTest from https://github.com/PhotonVision/photonvision/pull/1714 : 
Failure 1:

[==========] 52 tests from 7 test suites ran. (1505 ms total)
[  PASSED  ] 52 tests.
AddressSanitizer:DEADLYSIGNAL
=================================================================
==77094==ERROR: AddressSanitizer: SEGV on unknown address 0x000069800042 (pc 0x155550999d79 bp 0x1555499ff5a0 sp 0x1555499fed18 T9)
==77094==The signal is caused by a READ memory access.
    #0 0x155550999d79  (/lib/x86_64-linux-gnu/libc.so.6+0x199d79)
    #1 0x155554a942e9 in MemcmpInterceptorCommon(void*, int (*)(void const*, void const*, unsigned long), void const*, void const*, unsigned long) ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:881
    #2 0x155554a94bc6 in __interceptor_memcmp ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:892
    #3 0x155554a94bc6 in __interceptor_memcmp ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:887
    #4 0x15555549ac00 in std::char_traits<char>::compare(char const*, char const*, unsigned long) /usr/include/c++/11/bits/char_traits.h:389
    #5 0x15555549ac00 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const /usr/include/c++/11/bits/basic_string.tcc:1444
    #6 0x15555549ac00 in bool std::operator==<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*) /usr/include/c++/11/bits/basic_string.h:6250
    #7 0x15555549ac00 in GetSinkStreamValues /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:163
    #8 0x15555549ac00 in UpdateStreamValues /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:228
    #9 0x15555549ac00 in operator() /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:464
    #10 0x155554591e9e in std::function<void (cs::RawEvent const&)>::operator()(cs::RawEvent const&) const /usr/include/c++/11/bits/std_function.h:590
    #11 0x155554591e9e in cs::impl::NotifierThread::DoCallback(std::function<void (cs::RawEvent const&)>, cs::RawEvent const&) /work/cscore/src/main/native/cpp/Notifier.h:51
    #12 0x155554591e9e in wpi::CallbackThread<cs::impl::NotifierThread, cs::RawEvent, cs::impl::ListenerData, cs::RawEvent>::Main() /work/wpiutil/src/main/native/include/wpi/CallbackManager.h:165
    #13 0x1555537578da in operator() /work/wpiutil/src/main/native/cpp/SafeThread.cpp:81
    #14 0x1555537578da in __invoke_impl<void, wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:61
    #15 0x1555537578da in __invoke<wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:96
    #16 0x1555537578da in _M_invoke<0> /usr/include/c++/11/bits/std_thread.h:259
    #17 0x1555537578da in operator() /usr/include/c++/11/bits/std_thread.h:266
    #18 0x1555537578da in _M_run /usr/include/c++/11/bits/std_thread.h:211
    #19 0x1555514dc252  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
    #20 0x155550894ac2 in start_thread nptl/pthread_create.c:442
    #21 0x15555092684f  (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0x199d79) 
Thread T9 created by T0 here:
    #0 0x155554a58685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x1555514dc328 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc328)

==77094==ABORTING

Failure 2:

[==========] 52 tests from 7 test suites ran. (1515 ms total)
[  PASSED  ] 52 tests.
terminate called after throwing an instance of 'std::bad_function_call'
  what():  bad_function_call
./run-photonlibpy-to-crash.sh: line 31: 97288 Aborted                 "$PROGRAM"
(.venv) matt@photonvision:~/Documents/GitHub/photonvision$ 

Failure 3:

[==========] 52 tests from 7 test suites ran. (1529 ms total)
[  PASSED  ] 52 tests.
AddressSanitizer:DEADLYSIGNAL
=================================================================
==74081==ERROR: AddressSanitizer: SEGV on unknown address 0x00006980003b (pc 0x155554a2cc37 bp 0x00006980004b sp 0x1555499fed20 T9)
==74081==The signal is caused by a WRITE memory access.
    #0 0x155554a2cc37 in bool __sanitizer::atomic_compare_exchange_strong<__sanitizer::atomic_uint8_t>(__sanitizer::atomic_uint8_t volatile*, __sanitizer::atomic_uint8_t::Type*, __sanitizer::atomic_uint8_t::Type, __sanitizer::memory_order) ../../../../src/libsanitizer/sanitizer_common/sanitizer_atomic_clang.h:80
    #1 0x155554a2cc37 in __asan::Allocator::AtomicallySetQuarantineFlagIfAllocated(__asan::AsanChunk*, void*, __sanitizer::BufferedStackTrace*) ../../../../src/libsanitizer/asan/asan_allocator.cpp:621
    #2 0x155554a2cc37 in __asan::Allocator::Deallocate(void*, unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType) ../../../../src/libsanitizer/asan/asan_allocator.cpp:697
    #3 0x155554ab720c in operator delete(void*, unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:172
    #4 0x155555498f70 in __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) /usr/include/c++/11/ext/new_allocator.h:145
    #5 0x155555498f70 in std::allocator<char>::deallocate(char*, unsigned long) /usr/include/c++/11/bits/allocator.h:199
    #6 0x155555498f70 in std::allocator_traits<std::allocator<char> >::deallocate(std::allocator<char>&, char*, unsigned long) /usr/include/c++/11/bits/alloc_traits.h:496
    #7 0x155555498f70 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_destroy(unsigned long) /usr/include/c++/11/bits/basic_string.h:245
    #8 0x155555498f70 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose() /usr/include/c++/11/bits/basic_string.h:240
    #9 0x155555498f70 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() /usr/include/c++/11/bits/basic_string.h:672
    #10 0x155555498f70 in void std::destroy_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /usr/include/c++/11/bits/stl_construct.h:88
    #11 0x155555498f70 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /usr/include/c++/11/bits/stl_construct.h:149
    #12 0x155555498f70 in void std::_Destroy_aux<false>::__destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /usr/include/c++/11/bits/stl_construct.h:163
    #13 0x155555498f70 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /usr/include/c++/11/bits/stl_construct.h:196
    #14 0x155555498f70 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&) /usr/include/c++/11/bits/alloc_traits.h:848
    #15 0x155555498f70 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~vector() /usr/include/c++/11/bits/stl_vector.h:680
    #16 0x155555498f70 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_move_assign(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, std::integral_constant<bool, true>) /usr/include/c++/11/bits/stl_vector.h:1821
    #17 0x155555498f70 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::operator=(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&) /usr/include/c++/11/bits/stl_vector.h:714
    #18 0x155555498f70 in operator() /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:463
    #19 0x155554591e9e in std::function<void (cs::RawEvent const&)>::operator()(cs::RawEvent const&) const /usr/include/c++/11/bits/std_function.h:590
    #20 0x155554591e9e in cs::impl::NotifierThread::DoCallback(std::function<void (cs::RawEvent const&)>, cs::RawEvent const&) /work/cscore/src/main/native/cpp/Notifier.h:51
    #21 0x155554591e9e in wpi::CallbackThread<cs::impl::NotifierThread, cs::RawEvent, cs::impl::ListenerData, cs::RawEvent>::Main() /work/wpiutil/src/main/native/include/wpi/CallbackManager.h:165
    #22 0x1555537578da in operator() /work/wpiutil/src/main/native/cpp/SafeThread.cpp:81
    #23 0x1555537578da in __invoke_impl<void, wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:61
    #24 0x1555537578da in __invoke<wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:96
    #25 0x1555537578da in _M_invoke<0> /usr/include/c++/11/bits/std_thread.h:259
    #26 0x1555537578da in operator() /usr/include/c++/11/bits/std_thread.h:266
    #27 0x1555537578da in _M_run /usr/include/c++/11/bits/std_thread.h:211
    #28 0x1555514dc252  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
    #29 0x155550894ac2 in start_thread nptl/pthread_create.c:442
    #30 0x15555092684f  (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV ../../../../src/libsanitizer/sanitizer_common/sanitizer_atomic_clang.h:80 in bool __sanitizer::atomic_compare_exchange_strong<__sanitizer::atomic_uint8_t>(__sanitizer::atomic_uint8_t volatile*, __sanitizer::atomic_uint8_t::Type*, __sanitizer::atomic_uint8_t::Type, __sanitizer::memory_order)
Thread T9 created by T0 here:
    #0 0x155554a58685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x1555514dc328 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc328)

==74081==ABORTING

Failure 4:

[  PASSED  ] 52 tests.
AddressSanitizer:DEADLYSIGNAL
=================================================================
==89459==ERROR: AddressSanitizer: SEGV on unknown address 0x000069800050 (pc 0x155550999d79 bp 0x1555499ff5a0 sp 0x1555499fed18 T9)
==89459==The signal is caused by a READ memory access.
    #0 0x155550999d79  (/lib/x86_64-linux-gnu/libc.so.6+0x199d79)
    #1 0x155554a942e9 in MemcmpInterceptorCommon(void*, int (*)(void const*, void const*, unsigned long), void const*, void const*, unsigned long) ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:881
    #2 0x155554a94bc6 in __interceptor_memcmp ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:892
    #3 0x155554a94bc6 in __interceptor_memcmp ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:887
    #4 0x15555549ac00 in std::char_traits<char>::compare(char const*, char const*, unsigned long) /usr/include/c++/11/bits/char_traits.h:389
    #5 0x15555549ac00 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const /usr/include/c++/11/bits/basic_string.tcc:1444
    #6 0x15555549ac00 in bool std::operator==<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*) /usr/include/c++/11/bits/basic_string.h:6250
    #7 0x15555549ac00 in GetSinkStreamValues /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:163
    #8 0x15555549ac00 in UpdateStreamValues /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:228
    #9 0x15555549ac00 in operator() /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:464
    #10 0x155554591e9e in std::function<void (cs::RawEvent const&)>::operator()(cs::RawEvent const&) const /usr/include/c++/11/bits/std_function.h:590
    #11 0x155554591e9e in cs::impl::NotifierThread::DoCallback(std::function<void (cs::RawEvent const&)>, cs::RawEvent const&) /work/cscore/src/main/native/cpp/Notifier.h:51
    #12 0x155554591e9e in wpi::CallbackThread<cs::impl::NotifierThread, cs::RawEvent, cs::impl::ListenerData, cs::RawEvent>::Main() /work/wpiutil/src/main/native/include/wpi/CallbackManager.h:165
    #13 0x1555537578da in operator() /work/wpiutil/src/main/native/cpp/SafeThread.cpp:81
    #14 0x1555537578da in __invoke_impl<void, wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:61
    #15 0x1555537578da in __invoke<wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:96
    #16 0x1555537578da in _M_invoke<0> /usr/include/c++/11/bits/std_thread.h:259
    #17 0x1555537578da in operator() /usr/include/c++/11/bits/std_thread.h:266
    #18 0x1555537578da in _M_run /usr/include/c++/11/bits/std_thread.h:211
    #19 0x1555514dc252  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
    #20 0x155550894ac2 in start_thread nptl/pthread_create.c:442
    #21 0x15555092684f  (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0x199d79) 
Thread T9 created by T0 here:
    #0 0x155554a58685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x1555514dc328 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc328)

==89459==ABORTING
(.venv) matt@photonvision:~/Documents/GitHub/photonvision$

Based on the above asan logs, here, loos suspiscious to me. We're iterating over m_addresses from a function called by NotifierThread.

The cameraserver::instance could only come from this magic static here. The callback comes from m_videoListener here. The callback does capture this. Not sure if this setup could produce a race at destruction, since it seems like the VideoListener destructor deregisters the callback.

To Reproduce
Steps to reproduce the behavior:

  1. Checkout PhotonVision from 2025/01/12
  2. Run photon-lib unit tests repeatedly (for me, dozens of times per failure)
  3. Given sufficient runs, one will segfault eventually (implying this is a race)?
@mcm001 mcm001 added the type: bug Something isn't working. label Jan 15, 2025
@mcm001
Copy link
Contributor Author

mcm001 commented Jan 15, 2025 •
edited
Loading

See this:
https://github.com/wpilibsuite/allwpilib/blob/a14545102f2c16f8b446e765af8d872e028a92f8/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp#L83C1-L86C40

struct Instance {
  Instance();

//snip
  cs::VideoListener m_videoListener;
// snip
  std::vector<std::string> m_addresses;
};

Here, m_addresses will be destroyed before m_videoListener right? (source). And the callback that the video listener calls won't be unregistered until ~VideoListener.

@mcm001 mcm001 linked a pull request Jan 15, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type: bug Something isn't working.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[CameraServer] destroy VideoListener first mcm001/allwpilib
1 participant
@mcm001