velox_functions_remote_client_test is failing in Linux release with adapters CI

[majetideepak]

Bug description

[==========] Running 6 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 6 tests from RemoteFunctionTestFixture/RemoteFunctionTest
[ RUN ] RemoteFunctionTestFixture/RemoteFunctionTest.simple/0
I0922 22:44:36.204250 39419 ThriftServer.cpp:836] Using thread manager (resource pools not enabled) on address/port /tmp/socketYuJayP: runtime: thriftFlagNotSet, , thrift flag: false, enable gflag: false, disable gflag: false
*** Aborted at 1727045076 (Unix time, try 'date -d @1727045076') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 39348 (pthread TID 0x7f2505a00640) (linux TID 39438) (code: 128), stack trace: ***
(error retrieving stack trace)

System information

CI: https://github.com/facebookincubator/velox/actions/runs/10984888387/job/30496015047

Relevant logs

No response

[majetideepak]

I suspect the fbthrift upgrade is causing the test to fail.
CC: @czentgr

[czentgr]

This does look like the same issue we investigated for the C++20 standard update. This test started to SEGV consistently.
One of the solutions was to upgrade folly/fbthrift to the latest version - in a local build environment the test started to succeed again.

We will have to see if this a consistent failure and if it is test only (like the issue from the C++20).
Perhaps even upgrading to C++20 will fix this.

Of note is that the folly upgrade PR did not run into the issue but a subsequent PR did.

[majetideepak]

Of note is that the folly upgrade PR did not run into the issue but a subsequent PR did.

Because the new image build gets reflected on a subsequent build?

The failure seems consistent on main.

[czentgr]

Stack

Thread 30 "ThriftSrv-pri3-" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff86ffd640 (LWP 1892544)]
0x0000000008711923 in facebook::velox::functions::RemoteFunctionServiceHandler::~RemoteFunctionServiceHandler (this=0xffff0dfe50c8, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at /root/oss/velox/./velox/functions/remote/server/RemoteFunctionService.h:26
26      class RemoteFunctionServiceHandler
Missing separate debuginfos, use: dnf debuginfo-install gmock-1.11.0-1.el9.x86_64 gtest-1.11.0-1.el9.x86_64 libdwarf-0.3.4-1.el9.1.x86_64 libsodium-1.0.18-8.el9.x86_64 libunwind-1.6.2-1.el9.x86_64 re2-20211101-20.el9.x86_64
(gdb) bt
#0  0x0000000008711923 in facebook::velox::functions::RemoteFunctionServiceHandler::~RemoteFunctionServiceHandler (
    this=0xffff0dfe50c8, __in_chrg=<optimized out>, __vtt_parm=<optimized out>)
    at /root/oss/velox/./velox/functions/remote/server/RemoteFunctionService.h:26
#1  0x000000000886cd42 in apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}::operator()(folly::Unit) const ()
#2  0x0000000008875b89 in folly::futures::detail::wrapInvoke<folly::Unit, apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(folly::Try<folly::Unit>&&, apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&)::{lambda()#1}::operator()() const ()
#3  0x0000000008878ba9 in folly::SemiFuture<folly::Unit> folly::futures::detail::InvokeResultWrapperBase<folly::SemiFuture<folly::Unit> >::wrapResult<folly::futures::detail::wrapInvoke<folly::Unit, apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(folly::Try<folly::Unit>&&, apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&)::{lambda()#1}>(folly::futures::detail::wrapInvoke<folly::Unit, apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(folly::Try<folly::Unit>&&, apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&)::{lambda()#1}) ()
#4  0x0000000008875c06 in auto folly::futures::detail::wrapInvoke<folly::Unit, apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(folly::Try<folly::Unit>&&, apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) ()
#5  0x0000000008875c46 in folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}::operator()(folly::Try<folly::Unit>&&) ()
#6  0x000000000887ceb1 in folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}::operator()(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&) ()
#7  0x000000000887f33a in auto folly::futures::detail::CoreCallbackState<folly::Unit, folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}>::invoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<folly::Unit> >(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&) ()
#8  0x000000000887f373 in folly::futures::detail::CoreCallbackState<folly::Unit, folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}>::tryInvoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<folly::Unit> >(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)::{lambda()#1}::operator()() const ()
#9  0x000000000888280b in folly::makeTryWithNoUnwrap<folly::futures::detail::CoreCallbackState<folly::Unit, folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}>::tryInvoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<folly::Unit> >(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)::{lambda()#1}> ()
#10 0x00000000088816b6 in folly::makeTryWith<folly::futures::detail::CoreCallbackState<folly::Unit, folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}>::tryInvoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<folly::Unit> >(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)::{lambda()#1}> ()
#11 0x000000000887f3bd in auto folly::futures::detail::CoreCallbackState<folly::Unit, folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<f--Type <RET> for more, q to quit, c to continue without paging--
olly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}>::tryInvoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<folly::Unit> >(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&) ()
#12 0x000000000887f415 in folly::futures::detail::detail_msvc_15_7_workaround::tryInvoke<folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}>, folly::futures::detail::CoreCallbackState<folly::Unit, folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}>, folly::Unit, 0> ()
#13 0x000000000887f4d6 in folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}> >(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}>, folly::futures::detail::InlineContinuation)::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}::operator()(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&) ()
#14 0x00000000088831e5 in folly::futures::detail::Core<folly::Unit>::setCallback<folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}> >(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}>, folly::futures::detail::InlineContinuation)::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&, std::shared_ptr<folly::RequestContext>&&, folly::futures::detail::InlineContinuation)::{lambda(folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<folly::Executor>&&, folly::exception_wrapper*)#1}::operator()(folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<folly::Executor>&&, folly::exception_wrapper*) ()
#15 0x00000000088842bd in void folly::detail::function::call_<folly::futures::detail::Core<folly::Unit>::setCallback<folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAliv--Type <RET> for more, q to quit, c to continue without paging--
e<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}> >(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTryInline<folly::SemiFuture<folly::Unit>::deferValue<apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}>, folly::futures::detail::InlineContinuation)::{lambda(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<folly::Unit>&&)#1}>(apache::thrift::ThriftServer::callOnStartServing()::{lambda(folly::Unit)#1}&&, std::shared_ptr<folly::RequestContext>&&, folly::futures::detail::InlineContinuation)::{lambda(folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<folly::Executor>&&, folly::exception_wrapper*)#1}, true, false, void, folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<folly::Executor>&&, folly::exception_wrapper*>(folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<folly::Executor>&&, folly::exception_wrapper*, folly::detail::function::Data&) ()
#16 0x000000000f4ad854 in folly::detail::function::FunctionTraits<void (folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<folly::Executor>&&, folly::exception_wrapper*)>::operator()(folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<folly::Executor>&&, folly::exception_wrapper*) (this=0x7fffd40432f0, args#0=..., args#1=...,
    args#2=0x0) at /root/oss/velox/deps-download/folly/folly/Function.h:370
#17 0x000000000f4abd0c in operator() (__closure=0x7fffd4043670, ka=...)
    at /root/oss/velox/deps-download/folly/folly/futures/detail/Core.cpp:621
#18 0x000000000f4acc42 in folly::detail::function::call_<folly::futures::detail::CoreBase::doCallback(folly::Executor::KeepAlive<>&&, folly::futures::detail::State)::<lambda(folly::Executor::KeepAlive<>&&)>, true, false, void, folly::Executor::KeepAlive<folly::Executor>&&>(folly::detail::function::Data &) (p=...)
    at /root/oss/velox/deps-download/folly/folly/Function.h:341
#19 0x000000000f4ad11e in folly::detail::function::FunctionTraits<void (folly::Executor::KeepAlive<folly::Executor>&&)>::operator()(folly::Executor::KeepAlive<folly::Executor>&&) (this=0x7fffd4043670, args#0=...)
    at /root/oss/velox/deps-download/folly/folly/Function.h:370
#20 0x000000000f4ad15c in folly::Executor::KeepAlive<folly::Executor>::add<folly::Function<void (folly::Executor::KeepAlive<folly::Executor>&&)> >(folly::Function<void (folly::Executor::KeepAlive<folly::Executor>&&)>&&) &&::{lambda()#1}::operator()() (__closure=0x7fffd4043660) at /root/oss/velox/deps-download/folly/folly/Executor.h:187
#21 0x000000000f4ae005 in folly::detail::function::call_<folly::Executor::KeepAlive<folly::Executor>::add<folly::Function<void (folly::Executor::KeepAlive<folly::Executor>&&)> >(folly::Function<void (folly::Executor::KeepAlive<folly::Executor>&&)>&&) &&::{lambda()#1}, false, false, void>(, folly::detail::function::Data&) (p=...)
    at /root/oss/velox/deps-download/folly/folly/Function.h:341
#22 0x00000000086102ef in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7fffd40436d0)
    at /root/oss/deps/include/folly/Function.h:370
#23 0x00000000087ae8bf in apache::thrift::concurrency::FunctionRunner::run() ()
#24 0x0000000008dc2e17 in apache::thrift::concurrency::ThreadManager::Task::run()::{lambda()#1}::operator()() const
    ()
#25 0x0000000008dc2ea0 in apache::thrift::concurrency::ThreadManager::Task::run() ()
#26 0x0000000008dc49fb in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#27 0x0000000008db34fb in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#28 0x00007ffff6c89c52 in start_thread (arg=<optimized out>) at pthread_create.c:443
#29 0x00007ffff6d0ec80 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb) p $_siginfo
$1 = {si_signo = 11, si_errno = 0, si_code = 128, _sifields = {_pad = {0 <repeats 28 times>}, _kill = {si_pid = 0,
      si_uid = 0}, _timer = {si_tid = 0, si_overrun = 0, si_sigval = {sival_int = 0, sival_ptr = 0x0}}, _rt = {
      si_pid = 0, si_uid = 0, si_sigval = {sival_int = 0, sival_ptr = 0x0}}, _sigchld = {si_pid = 0, si_uid = 0,
      si_status = 0, si_utime = 0, si_stime = 0}, _sigfault = {si_addr = 0x0, _addr_lsb = 0, _addr_bnd = {
        _lower = 0x0, _upper = 0x0}}, _sigpoll = {si_band = 0, si_fd = 0}}}

This is different than the C++20 stack that prompted us to update folly/fbthrift.

[czentgr]

If we remove the virtual keyword for the RemoteFunctionServiceHandler when it inherits from remote::RemoteFunctionService the location for the SEGV changes. The destructor called with the virtual keyword is to disambiguate possible other inherited objects.

The new SEGV now occurs in the actual destructor of the created RemoteFunctionServiceHandler object in the test. It occurs after the std::string destructor for the functionPrefix_ member has been called and it is reducing the stack size that holds the std::string member. Apparently, this comes originally from a free call even though the destructor never explicitly calls free. This line of code is after adding an explicit destructor to be able to set a break point easily.
The destructor implementation is

  ~RemoteFunctionServiceHandler() {
    std::cout << "Test";
  };  <<< line 36

*** Aborted at 1727216285 (Unix time, try 'date -d @1727216285') ***
*** Signal 6 (SIGABRT) (0x1e1b78) received by PID 1973112 (pthread TID 0x7f5b5d7fa640) (linux TID 1973141) (maybe from PID 1973112, UID 0) (code: -6), stack trace: ***
    @ 00000000103214dc _ZN5folly10symbolizer12_GLOBAL__N_118innerSignalHandlerEiP9siginfo_tPv
                       /root/oss/velox/deps-download/folly/folly/debugging/symbolizer/SignalHandler.cpp:453
    @ 000000001032159b _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv
                       /root/oss/velox/deps-download/folly/folly/debugging/symbolizer/SignalHandler.cpp:474
    @ 000000000003e72f (unknown)
    @ 000000000008ba6c __pthread_kill_implementation
    @ 000000000003e685 raise
    @ 0000000000028832 abort
    @ 000000000002916f __libc_message.cold
    @ 0000000000095b16 malloc_printerr
    @ 00000000000973cb _int_free
    @ 0000000000099e54 free
    @ 0000000007dc86ec _ZN8facebook5velox9functions28RemoteFunctionServiceHandlerD2Ev
                       /root/oss/velox/./velox/functions/remote/server/RemoteFunctionService.h:36
                       -> /root/oss/velox/velox/functions/remote/server/RemoteFunctionService.cpp
    @ 0000000007dc6ecf _ZN8facebook5velox9functions28RemoteFunctionServiceHandlerD1Ev
                       /root/oss/velox/./velox/functions/remote/server/RemoteFunctionService.h:34
                       -> /root/oss/velox/velox/functions/remote/server/RemoteFunctionService.cpp
    @ 0000000007f4079b _ZZN6apache6thrift12ThriftServer18callOnStartServingEvENKUlN5folly4UnitEE_clES3_
                       /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp:1723
    @ 0000000007f495e2 _ZZN5folly7futures6detail10wrapInvokeINS_4UnitEZN6apache6thrift12ThriftServer18callOnStartServingEvEUlS3_E_EEDaONS_3TryIT_EEOT0_ENKUlvE_clEv
                       /root/oss/deps/include/folly/futures/Future-inl.h:92
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp
    @ 0000000007f4c602 _ZN5folly7futures6detail23InvokeResultWrapperBaseINS_10SemiFutureINS_4UnitEEEE10wrapResultIZNS1_10wrapInvokeIS4_ZN6apache6thrift12ThriftServer18callOnStartServingEvEUlS4_E_EEDaONS_3TryIT_EEOT0_EUlvE_EES5_SE_
                       /root/oss/deps/include/folly/futures/Future-inl.h:68
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp
    @ 0000000007f4965f _ZN5folly7futures6detail10wrapInvokeINS_4UnitEZN6apache6thrift12ThriftServer18callOnStartServingEvEUlS3_E_EEDaONS_3TryIT_EEOT0_
                       /root/oss/deps/include/folly/futures/Future-inl.h:98
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp
    @ 0000000007f4969f _ZZNO5folly10SemiFutureINS_4UnitEE10deferValueIZN6apache6thrift12ThriftServer18callOnStartServingEvEUlS1_E_EENS0_INS_7futures6detail19valueCallableResultIS1_T_E10value_typeEEEOSB_ENUlONS_3TryIS1_EEE_clESI_
                       /root/oss/deps/include/folly/futures/Future-inl.h:800
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp
    @ 0000000007f5090a _ZZNO5folly6FutureINS_4UnitEE13thenTryInlineIZNOS_10SemiFutureIS1_E10deferValueIZN6apache6thrift12ThriftServer18callOnStartServingEvEUlS1_E_EENS4_INS_7futures6detail19valueCallableResultIS1_T_E10value_typeEEEOSE_EUlONS_3TryIS1_EEE_EENS0_INSC_17tryCallableResultIS1_SE_E10value_typeEEESI_ENUlONS_8Executor9KeepAliveISR_EESL_E_clESU_SL_
                       /root/oss/deps/include/folly/futures/Future-inl.h:975
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp
    @ 0000000007f52d93 _ZN5folly7futures6detail17CoreCallbackStateINS_4UnitEZNOS_6FutureIS3_E13thenTryInlineIZNOS_10SemiFutureIS3_E10deferValueIZN6apache6thrift12ThriftServer18callOnStartServingEvEUlS3_E_EENS7_INS1_19valueCallableResultIS3_T_E10value_typeEEEOSF_EUlONS_3TryIS3_EEE_EENS4_INS1_17tryCallableResultIS3_SF_E10value_typeEEESJ_EUlONS_8Executor9KeepAliveISS_EESM_E_E6invokeIJSU_SL_EEEDaDpOT_
                       /root/oss/deps/include/folly/futures/Future-inl.h:136
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp
    @ 0000000007f52dcc _ZZN5folly7futures6detail17CoreCallbackStateINS_4UnitEZNOS_6FutureIS3_E13thenTryInlineIZNOS_10SemiFutureIS3_E10deferValueIZN6apache6thrift12ThriftServer18callOnStartServingEvEUlS3_E_EENS7_INS1_19valueCallableResultIS3_T_E10value_typeEEEOSF_EUlONS_3TryIS3_EEE_EENS4_INS1_17tryCallableResultIS3_SF_E10value_typeEEESJ_EUlONS_8Executor9KeepAliveISS_EESM_E_E9tryInvokeIJSU_SL_EEEDaDpOT_ENKUlvE_clEv
                       /root/oss/deps/include/folly/futures/Future-inl.h:141
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp
    @ 0000000007f56264 _ZN5folly19makeTryWithNoUnwrapIZNS_7futures6detail17CoreCallbackStateINS_4UnitEZNOS_6FutureIS4_E13thenTryInlineIZNOS_10SemiFutureIS4_E10deferValueIZN6apache6thrift12ThriftServer18callOnStartServingEvEUlS4_E_EENS8_INS2_19valueCallableResultIS4_T_E10value_typeEEEOSG_EUlONS_3TryIS4_EEE_EENS5_INS2_17tryCallableResultIS4_SG_E10value_typeEEESK_EUlONS_8Executor9KeepAliveIST_EESN_E_E9tryInvokeIJSV_SM_EEEDaDpOT_EUlvE_EENSt9enable_ifIXntsrSt7is_sameINS_13invoke_detail6traitsISG_E6resultIEEvE5valueENSL_IS1A_EEE4typeESK_
                       /root/oss/deps/include/folly/Try-inl.h:254
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp
    @ 0000000007f5510f _ZN5folly11makeTryWithIZNS_7futures6detail17CoreCallbackStateINS_4UnitEZNOS_6FutureIS4_E13thenTryInlineIZNOS_10SemiFutureIS4_E10deferValueIZN6apache6thrift12ThriftServer18callOnStartServingEvEUlS4_E_EENS8_INS2_19valueCallableResultIS4_T_E10value_typeEEEOSG_EUlONS_3TryIS4_EEE_EENS5_INS2_17tryCallableResultIS4_SG_E10value_typeEEESK_EUlONS_8Executor9KeepAliveIST_EESN_E_E9tryInvokeIJSV_SM_EEEDaDpOT_EUlvE_EENSt9enable_ifIXntsrNS_5isTryINS_13invoke_detail6traitsISG_E6resultIEEEE5valueENSL_IS1A_EEE4typeESK_
                       /root/oss/deps/include/folly/Try-inl.h:276
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp
    @ 0000000007f52e16 _ZN5folly7futures6detail17CoreCallbackStateINS_4UnitEZNOS_6FutureIS3_E13thenTryInlineIZNOS_10SemiFutureIS3_E10deferValueIZN6apache6thrift12ThriftServer18callOnStartServingEvEUlS3_E_EENS7_INS1_19valueCallableResultIS3_T_E10value_typeEEEOSF_EUlONS_3TryIS3_EEE_EENS4_INS1_17tryCallableResultIS3_SF_E10value_typeEEESJ_EUlONS_8Executor9KeepAliveISS_EESM_E_E9tryInvokeIJSU_SL_EEEDaDpOT_
                       /root/oss/deps/include/folly/futures/Future-inl.h:141
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp
    @ 0000000007f52e6e _ZN5folly7futures6detail27detail_msvc_15_7_workaround9tryInvokeINS1_25tryExecutorCallableResultINS_4UnitEZNOS_6FutureIS5_E13thenTryInlineIZNOS_10SemiFutureIS5_E10deferValueIZN6apache6thrift12ThriftServer18callOnStartServingEvEUlS5_E_EENS9_INS1_19valueCallableResultIS5_T_E10value_typeEEEOSH_EUlONS_3TryIS5_EEE_EENS6_INS1_17tryCallableResultIS5_SH_E10value_typeEEESL_EUlONS_8Executor9KeepAliveISU_EESO_E_EENS1_17CoreCallbackStateIS5_SY_EES5_Li0EEEDcSH_RT0_SX_ONSM_IT1_EE
                       /root/oss/deps/include/folly/futures/Future-inl.h:351
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp
    @ 0000000007f52f2f _ZZN5folly7futures6detail10FutureBaseINS_4UnitEE18thenImplementationIZNOS_6FutureIS3_E13thenTryInlineIZNOS_10SemiFutureIS3_E10deferValueIZN6apache6thrift12ThriftServer18callOnStartServingEvEUlS3_E_EENS9_INS1_19valueCallableResultIS3_T_E10value_typeEEEOSH_EUlONS_3TryIS3_EEE_EENS6_INS1_17tryCallableResultIS3_SH_E10value_typeEEESL_EUlONS_8Executor9KeepAliveISU_EESO_E_NS1_25tryExecutorCallableResultIS3_SY_EEEENSt9enable_ifIXsrNT0_13ReturnsFutureE5valueENS6_INS12_10value_typeEEEE4typeESL_S12_NS1_18InlineContinuationEENUlSX_SO_E_clESX_SO_
                       /root/oss/deps/include/folly/futures/Future-inl.h:446
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp
    @ 0000000007f56c3e _ZZN5folly7futures6detail4CoreINS_4UnitEE11setCallbackIZNS1_10FutureBaseIS3_E18thenImplementationIZNOS_6FutureIS3_E13thenTryInlineIZNOS_10SemiFutureIS3_E10deferValueIZN6apache6thrift12ThriftServer18callOnStartServingEvEUlS3_E_EENSC_INS1_19valueCallableResultIS3_T_E10value_typeEEEOSK_EUlONS_3TryIS3_EEE_EENS9_INS1_17tryCallableResultIS3_SK_E10value_typeEEESO_EUlONS_8Executor9KeepAliveISX_EESR_E_NS1_25tryExecutorCallableResultIS3_S11_EEEENSt9enable_ifIXsrNT0_13ReturnsFutureE5valueENS9_INS15_10value_typeEEEE4typeESO_S15_NS1_18InlineContinuationEEUlS10_SR_E_EEvSO_OSt10shared_ptrINS_14RequestContextEES1B_ENUlRNS1_8CoreBaseES10_PNS_17exception_wrapperEE_clES1I_S10_S1K_
                       /root/oss/deps/include/folly/futures/detail/Core.h:634
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp
    @ 0000000007f57d16 _ZN5folly6detail8function5call_IZNS_7futures6detail4CoreINS_4UnitEE11setCallbackIZNS4_10FutureBaseIS6_E18thenImplementationIZNOS_6FutureIS6_E13thenTryInlineIZNOS_10SemiFutureIS6_E10deferValueIZN6apache6thrift12ThriftServer18callOnStartServingEvEUlS6_E_EENSF_INS4_19valueCallableResultIS6_T_E10value_typeEEEOSN_EUlONS_3TryIS6_EEE_EENSC_INS4_17tryCallableResultIS6_SN_E10value_typeEEESR_EUlONS_8Executor9KeepAliveIS10_EESU_E_NS4_25tryExecutorCallableResultIS6_S14_EEEENSt9enable_ifIXsrNT0_13ReturnsFutureE5valueENSC_INS18_10value_typeEEEE4typeESR_S18_NS4_18InlineContinuationEEUlS13_SU_E_EEvSR_OSt10shared_ptrINS_14RequestContextEES1E_EUlRNS4_8CoreBaseES13_PNS_17exception_wrapperEE_Lb1ELb0EvJS1L_S13_S1N_EEET2_DpT3_RNS1_4DataE
                       /root/oss/deps/include/folly/Function.h:341
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/server/ThriftServer.cpp
    @ 000000001041fa3b _ZN5folly6detail8function14FunctionTraitsIFvRNS_7futures6detail8CoreBaseEONS_8Executor9KeepAliveIS7_EEPNS_17exception_wrapperEEEclES6_SA_SC_
                       /root/oss/velox/deps-download/folly/folly/Function.h:370
                       -> /root/oss/velox/deps-download/folly/folly/futures/detail/Core.cpp
    @ 000000001041def3 _ZZN5folly7futures6detail8CoreBase10doCallbackEONS_8Executor9KeepAliveIS3_EENS1_5StateEENUlS6_E0_clES6_
                       /root/oss/velox/deps-download/folly/folly/futures/detail/Core.cpp:621
    @ 000000001041ee29 _ZN5folly6detail8function5call_IZNS_7futures6detail8CoreBase10doCallbackEONS_8Executor9KeepAliveIS6_EENS4_5StateEEUlS9_E0_Lb1ELb0EvJS9_EEET2_DpT3_RNS1_4DataE
                       /root/oss/velox/deps-download/folly/folly/Function.h:341
                       -> /root/oss/velox/deps-download/folly/folly/futures/detail/Core.cpp
    @ 000000001041f305 _ZN5folly6detail8function14FunctionTraitsIFvONS_8Executor9KeepAliveIS3_EEEEclES6_
                       /root/oss/velox/deps-download/folly/folly/Function.h:370
                       -> /root/oss/velox/deps-download/folly/folly/futures/detail/Core.cpp
    @ 000000001041f343 _ZZNO5folly8Executor9KeepAliveIS0_E3addINS_8FunctionIFvOS2_EEEEEvOT_ENUlvE_clEv
                       /root/oss/velox/deps-download/folly/folly/Executor.h:187
                       -> /root/oss/velox/deps-download/folly/folly/futures/detail/Core.cpp
    @ 00000000104201ec _ZN5folly6detail8function5call_IZNOS_8Executor9KeepAliveIS3_E3addINS_8FunctionIFvOS5_EEEEEvOT_EUlvE_Lb0ELb0EvJEEET2_DpT3_RNS1_4DataE
                       /root/oss/velox/deps-download/folly/folly/Function.h:341
                       -> /root/oss/velox/deps-download/folly/folly/futures/detail/Core.cpp
    @ 0000000007ddfcb1 _ZN5folly6detail8function14FunctionTraitsIFvvEEclEv
                       /root/oss/deps/include/folly/Function.h:370
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/GeneratedCodeHelper.cpp
    @ 0000000007e77500 _ZN6apache6thrift11concurrency14FunctionRunner3runEv
                       /root/oss/velox/deps-download/fbthrift/./thrift/lib/cpp/concurrency/FunctionRunner.h:138
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp2/async/AsyncProcessor.cpp
    @ 00000000084a170a _ZZN6apache6thrift11concurrency13ThreadManager4Task3runEvENKUlvE_clEv
                       /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp/concurrency/ThreadManager.cpp:181
    @ 00000000084a1793 _ZN6apache6thrift11concurrency13ThreadManager4Task3runEv
                       /root/oss/deps/include/folly/lang/Exception.h:290
                       -> /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp/concurrency/ThreadManager.cpp
    @ 00000000084a32ee _ZN6apache6thrift11concurrency13ThreadManager4Impl6Worker3runEv
                       /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp/concurrency/ThreadManager.cpp:655
    @ 0000000008491dee _ZN6apache6thrift11concurrency13PthreadThread10threadMainEPv
                       /root/oss/velox/deps-download/fbthrift/thrift/lib/cpp/concurrency/PosixThreadFactory.cpp:210
    @ 0000000000089d21 start_thread
    @ 000000000010ed3f __clone3

Something appears to be clobbering the object stack.

[czentgr]

Since this was working fine in the previous version of folly/fbthrift I investigated when the test ran into an issue. I determined that the last known good level of folly/fbthrift is 2024.07.01.00. The next level 2024.07.08.00 starts getting SEGVs in the test. The stacks are different than what occurs using the latest level

[ RUN      ] RemoteFunctionTestFixture/RemoteFunctionTest.simple/0
I0924 22:38:10.257839 2043148 ThriftServer.cpp:1190] Resource pools check complete - allowed
I0924 22:38:10.258229 2043148 ThriftServer.cpp:1095] runtimeResourcePoolsChecks() is already completed and result is 1
I0924 22:38:10.258404 2043148 ThriftServer.cpp:841] Using thread manager (resource pools not enabled) on address/port /tmp/socketTuAukr: runtime: thriftFlagNotSet, , thrift flag: false, enable gflag: false, disable gflag: false
I0924 22:38:10.279569 2043148 ThriftServer.cpp:1452] QPS limit will be enforced by Thrift Server
*** Aborted at 1727242695 (Unix time, try 'date -d @1727242695') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 2043141 (pthread TID 0x7fa596ffd640) (linux TID 2043149) (code: 128), stack trace: ***
    @ 000000000f4c4ede _ZN5folly10symbolizer12_GLOBAL__N_118innerSignalHandlerEiP9siginfo_tPv
                       /root/oss/velox/deps-download/folly/folly/experimental/symbolizer/SignalHandler.cpp:453
    @ 000000000f4c4f9d _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv
                       /root/oss/velox/deps-download/folly/folly/experimental/symbolizer/SignalHandler.cpp:474
    @ 000000000003e72f (unknown)
    @ 000000000f46e930 _ZNKSt8functionIFvPvN5folly18TLPDestructionModeEEEclES0_S2_
                       /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/std_function.h:591
                       -> /root/oss/velox/deps-download/folly/folly/detail/ThreadLocalDetail.cpp
    @ 000000000f46ded1 _ZN5folly18threadlocal_detail14ElementWrapper7disposeENS_18TLPDestructionModeE
                       /root/oss/velox/deps-download/folly/folly/detail/ThreadLocalDetail.h:84
                       -> /root/oss/velox/deps-download/folly/folly/detail/ThreadLocalDetail.cpp
    @ 000000000f46c08f _ZN5folly18threadlocal_detail14StaticMetaBase12onThreadExitEPv
                       /root/oss/velox/deps-download/folly/folly/detail/ThreadLocalDetail.cpp:143
    @ 0000000000086e50 __GI___nptl_deallocate_tsd
    @ 0000000000089bf5 start_thread
    @ 000000000010ed3f __clone3
Segmentation fault (core dumped)

There might be subsequent levels that work fine again and a subsequent change leads to bad behavior.
However, we also determined that the 2024.07.01.00 level bridges the gap because the previous level caused SEGV when compiled with C++20 standard. But both C++17 and C++20 do not cause SEGVs in the velox_functions_remote_client_test. Hopefully, this was the only problem and nothing new pops up when attempting to use 2024.07.01.00 level.

So we will be downgrading (but upgrading from the original level) from the currently used latest level to 2024.07.01.00.

[pedroerp]

I suspect this is because with this build we link both Parquet and remote function support to the same binary, and one brings thrift as a dependency; the other brings fbthrift.

I remember in the past having problems with this since they add symbols to the same namespace and end up confusing the linker and causing all sorts of weird crashes. I thought I had added a check somewhere, but we shouldn't allow customers to enable these options simultaneously.

[pedroerp]

As a test, could you try to disable Parquet support in this build and run the remote function test again?

[majetideepak]

@pedroerp One of the challenges now is that the Parquet->arrow->thrift dependency is installed as part of the build image.
Before Arrow was mainly bundled and disabling Parquet would disable thrift as well.
I removed regular thrift from the linking. I still saw a SEGV.
Another problem could be that the regular thrift compiler was used to generate the CPP files. But Christian validated that this was not the root cause as well.

[pedroerp]

I removed regular thrift from the linking.

Ouch. Did you validate it isn't getting pulled in somehow using ldd and nm?

[majetideepak]

Did you validate it isn't getting pulled in somehow using ldd and nm?

Both Thrift and FBThrift were always linked statically. The order of linkage caused issues previously.
I will confirm with ldd and nm.

The strange part is that it only happens with newer FBThrift versions as Chrisitan posted.

[pedroerp]

The strange part is that it only happens with newer FBThrift versions as Chrisitan posted.

It could be quite nondeterministic though. Maybe a symbol got renamed and that triggered a different behavior in the linker and exposed this issue. I remember when I was debugging this problem in Prestissimo, it was pretty hard to pinpoint what was happening.

[czentgr]

Right, we checked and saw that libthrift was linked last. This means it would only link in symbols from there if they hadn't been satisfied by the fbthrift libraries. So unless some symbol definition is missing from fbthrift no symbols from libthrift should be used.
When we removed libthrift manually from the link step we didn't encounter link issues. So all symbols should have been defined by fbthrift.

The interesting thing about this is that compiling with C++20 level and the latest fbthrift no problems occur. That is why we upgraded to the latest in the first place. Meanwhile, using C++20 on the previous old fbthrift level causes a SEGV too. So something gets tangled somehow (more details are in the C++20 PR).

But I think there is one level that works for both (C++17 and C++20) which is one that is now targeted with PR #11104.

This time we are testing with new container images that uses this level of FBOS.