Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage crash when leader change frequently under high load. #2889

Closed
kikimo opened this issue Sep 17, 2021 · 1 comment
Closed

storage crash when leader change frequently under high load. #2889

kikimo opened this issue Sep 17, 2021 · 1 comment
Assignees
@liuyu85cn
Labels
type/bug Type: something is unexpected
Milestone
v2.6.0

Comments

@kikimo
Copy link
Contributor

kikimo commented Sep 17, 2021 •
edited
Loading

Please check the FAQ documentation before raising an issue

Please check the FAQ documentation and old issues before raising an issue in case someone has asked the same question that you are asking.

Describe the bug (must be provided)

storage crash when leader change frequently under high load.

Your Environments (must be provided)

  • OS: uname -a
  • Compliler: g++ --version or clang++ --version
  • CPU: lscpu
  • Commit id (e.g. a3ffc7d8)

How To Reproduce(must be provided)

Steps to reproduce the behavior:

  1. cluster with 3storage + 3graph + 1meta
  2. insert edge concurrently
  3. keep stopping one of the graph storage randomly(you can achieve this bu sending SIGSTOP/SIGSTOP to storaged instance)

the corrupt log:

*** Aborted at 1631859979 (Unix time, try 'date -d @1631859979') ***
*** Signal 11 (SIGSEGV) (0x28) received by PID 68035 (pthread TID 0x7fffaadff700) (linux TID 69753) (code: address not mapped to object), stack trace: ***
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly10symbolizer17getStackTraceSafeEPmm+0x31)[0x4413c81]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x1b)[0x44091bb]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x4406f12]
/lib64/libpthread.so.0(+0xf62f)[0x7ffff74ae62f]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN6nebula15MemoryLockGuardINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEE11forceUnlockEv+0xa)[0x2afc45a]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN6nebula7storage27ChainAddEdgesProcessorLocal16markDanglingEdgeEv+0x24)[0x2af9266]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x2af9a22]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x2afc52f]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x2afd2b7]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x2afc5b2]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x2afc5fc]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x2afd377]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x2afd3d5]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x2afd432]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x2afed70]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x2afd50f]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x2aff99a]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x2b00261]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x436829b]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly7futures6detail8CoreBase10doCallbackEONS_8Executor9KeepAliveIS3_EENS1_5StateE+0x321)[0x4369101]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly7futures6detail4CoreIN6nebula7storage4cpp212ExecResponseEE9setResultEONS_8Executor9KeepAliveIS8_EEONS_3TryIS6_EE+0x63)[0x2934a45]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly7futures6detail4CoreIN6nebula7storage4cpp212ExecResponseEE9setResultEONS_3TryIS6_EE+0x4a)[0x2933f9a]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly7PromiseIN6nebula7storage4cpp212ExecResponseEE6setTryEONS_3TryIS4_EE+0x3a)[0x29377ee]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly7PromiseIN6nebula7storage4cpp212ExecResponseEE8setValueIS4_EEvOT_+0x36)[0x29371ca]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN6nebula7storage13BaseProcessorINS0_4cpp212ExecResponseEE10onFinishedEv+0xfe)[0x2936ee0]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN6nebula7storage13BaseProcessorINS0_4cpp212ExecResponseEE11handleAsyncEiiNS_4cpp29ErrorCodeE+0x1ae)[0x29567a0]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x297db9d]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x2985e14]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly6detail8function14FunctionTraitsIFvN6nebula4cpp29ErrorCodeEEEclES5_+0x2f)[0x3a9069b]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a8b028]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a9094e]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a92cdb]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a909d1]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a90a1a]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a92d9b]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a92df9]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a92e56]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a96f6a]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a92f33]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a9a12f]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a9d7ef]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly7futures6detail8CoreBase10doCallbackEONS_8Executor9KeepAliveIS3_EENS1_5StateE+0x397)[0x4369177]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly7futures6detail8CoreBase12setCallback_EONS_8FunctionIFvRS2_ONS_8Executor9KeepAliveIS5_EEPNS_17exception_wrapperEEEEOSt10shared_ptrINS_14RequestContextEENS1_18InlineContinuationE+0xde)[0x43698fe]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a9a204]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a9712b]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a930e0]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x3a90ae4]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN6nebula7kvstore4Part16asyncAppendBatchEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEN5folly8FunctionIFvNS_4cpp29ErrorCodeEEEE+0xa1)[0x3a8b0f1]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN6nebula7kvstore11NebulaStore16asyncAppendBatchEiiONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEN5folly8FunctionIFvNS_4cpp29ErrorCodeEEEE+0xef)[0x3ac612f]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN6nebula7storage17AddEdgesProcessor9doProcessERKNS0_4cpp215AddEdgesRequestE+0x107b)[0x297ec1f]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN6nebula7storage17AddEdgesProcessor7processERKNS0_4cpp215AddEdgesRequestE+0x627)[0x297d963]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN6nebula7storage27ChainAddEdgesProcessorLocal26forwardToDelegateProcessorEv+0x115)[0x2af9b97]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN6nebula7storage22ResumeAddEdgeProcessor12processLocalENS_4cpp29ErrorCodeE+0x363)[0x2b03853]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZZZN6nebula7storage18TransactionManager12addChainTaskEPNS0_18ChainBaseProcessorEENKUlvE_clEvENKUlOT_E0_clINS_4cpp29ErrorCodeEEEDaS6_+0x3d)[0x2ae1307]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZZN5folly7futures6detail10wrapInvokeIN6nebula4cpp29ErrorCodeEZZNS3_7storage18TransactionManager12addChainTaskEPNS6_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EEDaONS_3TryISB_EEOT0_ENKUlvE_clEv+0x37)[0x2ae3ee1]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly7futures6detail23InvokeResultWrapperBaseINS_10SemiFutureIN6nebula4cpp29ErrorCodeEEEE10wrapResultIZNS1_10wrapInvokeIS6_ZZNS4_7storage18TransactionManager12addChainTaskEPNSB_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EEDaONS_3TryISG_EEOT0_EUlvE_EES7_SG_+0x2d)[0x2ae542d]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly7futures6detail10wrapInvokeIN6nebula4cpp29ErrorCodeEZZNS3_7storage18TransactionManager12addChainTaskEPNS6_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EEDaONS_3TryISB_EEOT0_+0x7b)[0x2ae3f68]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZZNO5folly6FutureIN6nebula4cpp29ErrorCodeEE9thenValueIZZNS1_7storage18TransactionManager12addChainTaskEPNS6_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENS0_INS_7futures6detail19valueCallableResultIS3_SB_E10value_typeEEESC_ENUlONS_8Executor9KeepAliveISK_EEONS_3TryIS3_EEE_clESN_SQ_+0x3e)[0x2ae3e9e]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly7futures6detail17CoreCallbackStateIN6nebula4cpp29ErrorCodeEZNOS_6FutureIS5_E9thenValueIZZNS3_7storage18TransactionManager12addChainTaskEPNS9_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENS6_INS1_19valueCallableResultIS5_SE_E10value_typeEEESF_EUlONS_8Executor9KeepAliveISL_EEONS_3TryIS5_EEE_E6invokeIJSN_SQ_EEEDaDpOT_+0x5b)[0x2ae5783]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZZN5folly7futures6detail17CoreCallbackStateIN6nebula4cpp29ErrorCodeEZNOS_6FutureIS5_E9thenValueIZZNS3_7storage18TransactionManager12addChainTaskEPNS9_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENS6_INS1_19valueCallableResultIS5_SE_E10value_typeEEESF_EUlONS_8Executor9KeepAliveISL_EEONS_3TryIS5_EEE_E9tryInvokeIJSN_SQ_EEEDaDpOT_ENKUlvE_clEv+0x30)[0x2ae571c]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly19makeTryWithNoUnwrapIZNS_7futures6detail17CoreCallbackStateIN6nebula4cpp29ErrorCodeEZNOS_6FutureIS6_E9thenValueIZZNS4_7storage18TransactionManager12addChainTaskEPNSA_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENS7_INS2_19valueCallableResultIS6_SF_E10value_typeEEESG_EUlONS_8Executor9KeepAliveISM_EEONS_3TryIS6_EEE_E9tryInvokeIJSO_SR_EEEDaDpOT_EUlvE_EENSt9enable_ifIXntsrSt7is_sameINS_13invoke_detail6traitsISF_E6resultIEEvE5valueENSQ_IS16_EEE4typeESG_+0x20)[0x2ae8bdc]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly11makeTryWithIZNS_7futures6detail17CoreCallbackStateIN6nebula4cpp29ErrorCodeEZNOS_6FutureIS6_E9thenValueIZZNS4_7storage18TransactionManager12addChainTaskEPNSA_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENS7_INS2_19valueCallableResultIS6_SF_E10value_typeEEESG_EUlONS_8Executor9KeepAliveISM_EEONS_3TryIS6_EEE_E9tryInvokeIJSO_SR_EEEDaDpOT_EUlvE_EENSt9enable_ifIXntsrNS_5isTryINS_13invoke_detail6traitsISF_E6resultIEEEE5valueENSQ_IS16_EEE4typeESG_+0x2b)[0x2ae71b2]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly7futures6detail17CoreCallbackStateIN6nebula4cpp29ErrorCodeEZNOS_6FutureIS5_E9thenValueIZZNS3_7storage18TransactionManager12addChainTaskEPNS9_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENS6_INS1_19valueCallableResultIS5_SE_E10value_typeEEESF_EUlONS_8Executor9KeepAliveISL_EEONS_3TryIS5_EEE_E9tryInvokeIJSN_SQ_EEEDaDpOT_+0x48)[0x2ae57d6]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly7futures6detail27detail_msvc_15_7_workaround9tryInvokeINS1_25tryExecutorCallableResultIN6nebula4cpp29ErrorCodeEZNOS_6FutureIS7_E9thenValueIZZNS5_7storage18TransactionManager12addChainTaskEPNSB_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENS8_INS1_19valueCallableResultIS7_SG_E10value_typeEEESH_EUlONS_8Executor9KeepAliveISN_EEONS_3TryIS7_EEE_vEENS1_17CoreCallbackStateIS7_SU_EES7_Li0EEEDcSG_RT0_SQ_ONSR_IT1_EE+0x53)[0x2ae5834]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZZN5folly7futures6detail10FutureBaseIN6nebula4cpp29ErrorCodeEE18thenImplementationIZNOS_6FutureIS5_E9thenValueIZZNS3_7storage18TransactionManager12addChainTaskEPNSB_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENS8_INS1_19valueCallableResultIS5_SG_E10value_typeEEESH_EUlONS_8Executor9KeepAliveISN_EEONS_3TryIS5_EEE_NS1_25tryExecutorCallableResultIS5_SU_vEEEENSt9enable_ifIXsrNT0_13ReturnsFutureE5valueENSY_6ReturnEE4typeESH_SY_NS1_18InlineContinuationEENUlSQ_ST_E_clESQ_ST_+0xc9)[0x2ae5543]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZZN5folly7futures6detail4CoreIN6nebula4cpp29ErrorCodeEE11setCallbackIZNS1_10FutureBaseIS5_E18thenImplementationIZNOS_6FutureIS5_E9thenValueIZZNS3_7storage18TransactionManager12addChainTaskEPNSE_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENSB_INS1_19valueCallableResultIS5_SJ_E10value_typeEEESK_EUlONS_8Executor9KeepAliveISQ_EEONS_3TryIS5_EEE_NS1_25tryExecutorCallableResultIS5_SX_vEEEENSt9enable_ifIXsrNT0_13ReturnsFutureE5valueENS11_6ReturnEE4typeESK_S11_NS1_18InlineContinuationEEUlST_SW_E_EEvSK_OSt10shared_ptrINS_14RequestContextEES16_ENUlRNS1_8CoreBaseEST_PNS_17exception_wrapperEE_clES1D_ST_S1F_+0xc9)[0x2ae8da5]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly6detail8function14FunctionTraitsIFvRNS_7futures6detail8CoreBaseEONS_8Executor9KeepAliveIS7_EEPNS_17exception_wrapperEEE9callSmallIZNS4_4CoreIN6nebula4cpp29ErrorCodeEE11setCallbackIZNS4_10FutureBaseISJ_E18thenImplementationIZNOS_6FutureISJ_E9thenValueIZZNSH_7storage18TransactionManager12addChainTaskEPNSS_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENSP_INS4_19valueCallableResultISJ_SX_E10value_typeEEESY_EUlSA_ONS_3TryISJ_EEE_NS4_25tryExecutorCallableResultISJ_S17_vEEEENSt9enable_ifIXsrNT0_13ReturnsFutureE5valueENS1B_6ReturnEE4typeESY_S1B_NS4_18InlineContinuationEEUlSA_S16_E_EEvSY_OSt10shared_ptrINS_14RequestContextEES1G_EUlS6_SA_SC_E_EEvS6_SA_SC_RNS1_4DataE+0x3e)[0x2aeb94a]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x436829b]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly18ThreadPoolExecutor7runTaskERKSt10shared_ptrINS0_6ThreadEEONS0_4TaskE+0xa5)[0x43444e5]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x43375ab]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly23AtomicNotificationQueueINS_8FunctionIFvvEEEE5driveIRNS_9EventBase10FuncRunnerEEEbOT_+0x9a)[0x43b069a]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZThn40_N5folly32EventBaseAtomicNotificationQueueINS_8FunctionIFvvEEENS_9EventBase10FuncRunnerEE12handlerReadyEt+0x2c)[0x43b180c]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x4481eee]
/root/src/liuyu/nebula/build/bin/nebula-storaged(event_base_loop+0x50e)[0x448280e]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly9EventBase8loopBodyEib+0x95d)[0x43ac55d]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly9EventBase4loopEv+0x35)[0x43aca05]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly9EventBase11loopForeverEv+0x15)[0x43ae4b5]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly20IOThreadPoolExecutor9threadRunESt10shared_ptrINS_18ThreadPoolExecutor6ThreadEE+0x388)[0x43380e8]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly6detail8function14FunctionTraitsIFvvEE7callBigISt5_BindIFMNS_18ThreadPoolExecutorEFvSt10shared_ptrINS7_6ThreadEEEPS7_SA_EEEEvRNS1_4DataE+0x48)[0x4345cb8]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZN5folly6detail8function14FunctionTraitsIFvvEEclEv+0x25)[0x28cf5f9]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZZN5folly18NamedThreadFactory9newThreadEONS_8FunctionIFvvEEEENUlvE_clEv+0x43)[0x28deebd]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZSt13__invoke_implIvZN5folly18NamedThreadFactory9newThreadEONS0_8FunctionIFvvEEEEUlvE_JEET_St14__invoke_otherOT0_DpOT1_+0x1d)[0x28eed05]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZSt8__invokeIZN5folly18NamedThreadFactory9newThreadEONS0_8FunctionIFvvEEEEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS8_DpOS9_+0x23)[0x28e8dc1]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZNSt6thread8_InvokerISt5tupleIJZN5folly18NamedThreadFactory9newThreadEONS2_8FunctionIFvvEEEEUlvE_EEE9_M_invokeIJLm0EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE+0x25)[0x2929a4d]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZNSt6thread8_InvokerISt5tupleIJZN5folly18NamedThreadFactory9newThreadEONS2_8FunctionIFvvEEEEUlvE_EEEclEv+0x1a)[0x2928f4e]
/root/src/liuyu/nebula/build/bin/nebula-storaged(_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5folly18NamedThreadFactory9newThreadEONS3_8FunctionIFvvEEEEUlvE_EEEEE6_M_runEv+0x19)[0x2926207]
/root/src/liuyu/nebula/build/bin/nebula-storaged[0x483603e]
/lib64/libpthread.so.0(+0x7ea4)[0x7ffff74a6ea4]
/lib64/libc.so.6(clone+0x6c)[0x7ffff71cf9fc]
(safe mode, symbolizer not available)
ASAN:DEADLYSIGNAL
=================================================================
==85851==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000028 (pc 0x000002ee9c9b bp 0x7fff7acd2c00 sp 0x7fff7acd2bf0 T143)
==85851==The signal is caused by a WRITE memory access.
==85851==Hint: address points to the zero page.
ASAN:DEADLYSIGNAL
==85851==AddressSanitizer: while reporting a bug found another one. Ignoring.
    #0 0x2ee9c9a in nebula::MemoryLockGuard<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::forceUnlock() (build/bin/nebula-storaged+0x2ee9c9a)
    #1 0x2edf5b7 in nebula::storage::ChainAddEdgesProcessorLocal::markDanglingEdge() (build/bin/nebula-storaged+0x2edf5b7)
    #2 0x2ee0ffb in _ZZN6nebula7storage27ChainAddEdgesProcessorLocal26forwardToDelegateProcessorEvENUlOT_E0_clINS0_4cpp212ExecResponseEEEDaS3_ (build/bin/nebula-storaged+0x2ee0ffb)
    #3 0x2ee9e76 in auto folly::futures::detail::wrapInvoke<nebula::storage::cpp2::ExecResponse, nebula::storage::ChainAddEdgesProcessorLocal::forwardToDelegateProcessor()::{lambda(auto:1&&)#2}>(folly::Try<nebula::storage::cpp2::ExecResponse>&&, nebula::storage::ChainAddEdgesProcessorLocal::forwardToDelegateProcessor()::{lambda(auto:1&&)#2}&&)::{lambda()#1}::operator()() const (build/bin/nebula-storaged+0x2ee9e76)
    #4 0x2eebdf7 in nebula::storage::ChainAddEdgesProcessorLocal::forwardToDelegateProcessor()::{lambda(auto:1&&)#2}<folly::Unit> folly::futures::detail::InvokeResultWrapper<void>::wrapResult<auto folly::futures::detail::wrapInvoke<nebula::storage::cpp2::ExecResponse, nebula::storage::ChainAddEdgesProcessorLocal::forwardToDelegateProcessor()::{lambda(auto:1&&)#2}>(folly::Try<nebula::storage::cpp2::ExecResponse>&&, nebula::storage::ChainAddEdgesProcessorLocal::forwardToDelegateProcessor()::{lambda(auto:1&&)#2}&&)::{lambda()#1}>(auto folly::futures::detail::wrapInvoke<nebula::storage::cpp2::ExecResponse, nebula::storage::ChainAddEdgesProcessorLocal::forwardToDelegateProcessor()::{lambda(auto:1&&)#2}>(folly::Try<nebula::storage::cpp2::ExecResponse>&&, nebula::storage::ChainAddEdgesProcessorLocal::forwardToDelegateProcessor()::{lambda(auto:1&&)#2}&&)::{lambda()#1}) (build/bin/nebula-storaged+0x2eebdf7)
    #5 0x2eea0bd in auto folly::futures::detail::wrapInvoke<nebula::storage::cpp2::ExecResponse, nebula::storage::ChainAddEdgesProcessorLocal::forwardToDelegateProcessor()::{lambda(auto:1&&)#2}>(folly::Try<nebula::storage::cpp2::ExecResponse>&&, nebula::storage::ChainAddEdgesProcessorLocal::forwardToDelegateProcessor()::{lambda(auto:1&&)#2}&&) (build/bin/nebula-storaged+0x2eea0bd)
    #6 0x2eea1b9 in _ZZNO5folly6FutureIN6nebula7storage4cpp212ExecResponseEE9thenValueIZNS2_27ChainAddEdgesProcessorLocal26forwardToDelegateProcessorEvEUlOT_E0_EENS0_INS_7futures6detail19valueCallableResultIS4_S8_E10value_typeEEES9_ENUlONS_8Executor9KeepAliveISH_EEONS_3TryIS4_EEE_clESK_SN_ (build/bin/nebula-storaged+0x2eea1b9)
    #7 0x2eebf19 in auto folly::futures::detail::CoreCallbackState<folly::Unit, folly::Future<folly::futures::detail::valueCallableResult<nebula::storage::cpp2::ExecResponse, nebula::storage::ChainAddEdgesProcessorLocal::forwardToDelegateProcessor()::{lambda(auto:1&&)#2}>::value_type> folly::Future<nebula::storage::cpp2::ExecResponse>::thenValue<nebula::storage::ChainAddEdgesProcessorLocal::forwardToDelegateProcessor()::{lambda(auto:1&&)#2}>(nebula::storage::ChainAddEdgesProcessorLocal::forwardToDelegateProcessor()::{lambda(auto:1&&)#2}&&) &&::{lambda(folly::Executor::KeepAlive<folly::Future<folly::futures::detail::valueCallableResult<nebula::storage::cpp2::ExecResponse, auto:1>::value_type> >&&, folly::Try<nebula::storage::cpp2::ExecResponse>&&)#1}>::invoke<folly::Executor::KeepAlive, folly::Try>(folly::Executor::KeepAlive&&, folly::Try&&) (build/bin/nebula-storaged+0x2eebf19)
    #8 0x2eebfda in _ZN5folly7futures6detail27detail_msvc_15_7_workaround6invokeINS1_25tryExecutorCallableResultIN6nebula7storage4cpp212ExecResponseEZNOS_6FutureIS8_E9thenValueIZNS6_27ChainAddEdgesProcessorLocal26forwardToDelegateProcessorEvEUlOT_E0_EENS9_INS1_19valueCallableResultIS8_SD_E10value_typeEEESE_EUlONS_8Executor9KeepAliveISK_EEONS_3TryIS8_EEE_vEENS1_17CoreCallbackStateINS_4UnitESR_EES8_Li0EEEDcSD_RT0_SN_ONSO_IT1_EE (build/bin/nebula-storaged+0x2eebfda)
    #9 0x2eec10e in _ZZZN5folly7futures6detail10FutureBaseIN6nebula7storage4cpp212ExecResponseEE18thenImplementationIZNOS_6FutureIS6_E9thenValueIZNS4_27ChainAddEdgesProcessorLocal26forwardToDelegateProcessorEvEUlOT_E0_EENS9_INS1_19valueCallableResultIS6_SD_E10value_typeEEESE_EUlONS_8Executor9KeepAliveISK_EEONS_3TryIS6_EEE_NS1_25tryExecutorCallableResultIS6_SR_vEEEENSt9enable_ifIXntsrNT0_13ReturnsFutureE5valueENSV_6ReturnEE4typeESE_SV_NS1_18InlineContinuationEENUlSN_SQ_E_clESN_SQ_ENKUlvE_clEv (build/bin/nebula-storaged+0x2eec10e)
    #10 0x2eefc40 in _ZN5folly11makeTryWithIZZNS_7futures6detail10FutureBaseIN6nebula7storage4cpp212ExecResponseEE18thenImplementationIZNOS_6FutureIS7_E9thenValueIZNS5_27ChainAddEdgesProcessorLocal26forwardToDelegateProcessorEvEUlOT_E0_EENSA_INS2_19valueCallableResultIS7_SE_E10value_typeEEESF_EUlONS_8Executor9KeepAliveISL_EEONS_3TryIS7_EEE_NS2_25tryExecutorCallableResultIS7_SS_vEEEENSt9enable_ifIXntsrNT0_13ReturnsFutureE5valueENSW_6ReturnEE4typeESF_SW_NS2_18InlineContinuationEENUlSO_SR_E_clESO_SR_EUlvE_EENSV_IXsrNS_5isTryINS_13invoke_detail6traitsISE_E6resultIEEEE5valueES19_E4typeESF_ (build/bin/nebula-storaged+0x2eefc40)
    #11 0x2eec397 in _ZZN5folly7futures6detail10FutureBaseIN6nebula7storage4cpp212ExecResponseEE18thenImplementationIZNOS_6FutureIS6_E9thenValueIZNS4_27ChainAddEdgesProcessorLocal26forwardToDelegateProcessorEvEUlOT_E0_EENS9_INS1_19valueCallableResultIS6_SD_E10value_typeEEESE_EUlONS_8Executor9KeepAliveISK_EEONS_3TryIS6_EEE_NS1_25tryExecutorCallableResultIS6_SR_vEEEENSt9enable_ifIXntsrNT0_13ReturnsFutureE5valueENSV_6ReturnEE4typeESE_SV_NS1_18InlineContinuationEENUlSN_SQ_E_clESN_SQ_ (build/bin/nebula-storaged+0x2eec397)
    #12 0x2ef119a in _ZZN5folly7futures6detail4CoreIN6nebula7storage4cpp212ExecResponseEE11setCallbackIZNS1_10FutureBaseIS6_E18thenImplementationIZNOS_6FutureIS6_E9thenValueIZNS4_27ChainAddEdgesProcessorLocal26forwardToDelegateProcessorEvEUlOT_E0_EENSC_INS1_19valueCallableResultIS6_SG_E10value_typeEEESH_EUlONS_8Executor9KeepAliveISN_EEONS_3TryIS6_EEE_NS1_25tryExecutorCallableResultIS6_SU_vEEEENSt9enable_ifIXntsrNT0_13ReturnsFutureE5valueENSY_6ReturnEE4typeESH_SY_NS1_18InlineContinuationEEUlSQ_ST_E_EEvSH_OSt10shared_ptrINS_14RequestContextEES13_ENUlRNS1_8CoreBaseESQ_PNS_17exception_wrapperEE_clES1A_SQ_S1C_ (build/bin/nebula-storaged+0x2ef119a)
    #13 0x2ef285a in _ZN5folly6detail8function14FunctionTraitsIFvRNS_7futures6detail8CoreBaseEONS_8Executor9KeepAliveIS7_EEPNS_17exception_wrapperEEE9callSmallIZNS4_4CoreIN6nebula7storage4cpp212ExecResponseEE11setCallbackIZNS4_10FutureBaseISK_E18thenImplementationIZNOS_6FutureISK_E9thenValueIZNSI_27ChainAddEdgesProcessorLocal26forwardToDelegateProcessorEvEUlOT_E0_EENSQ_INS4_19valueCallableResultISK_SU_E10value_typeEEESV_EUlSA_ONS_3TryISK_EEE_NS4_25tryExecutorCallableResultISK_S14_vEEEENSt9enable_ifIXntsrNT0_13ReturnsFutureE5valueENS18_6ReturnEE4typeESV_S18_NS4_18InlineContinuationEEUlSA_S13_E_EEvSV_OSt10shared_ptrINS_14RequestContextEES1D_EUlS6_SA_SC_E_EEvS6_SA_SC_RNS1_4DataE (build/bin/nebula-storaged+0x2ef285a)
    #14 0x5e6dbbb  (build/bin/nebula-storaged+0x5e6dbbb)
    #15 0x5e6ea21 in folly::futures::detail::CoreBase::doCallback(folly::Executor::KeepAlive<folly::Executor>&&, folly::futures::detail::State) (build/bin/nebula-storaged+0x5e6ea21)
    #16 0x2acab30 in folly::futures::detail::Core<nebula::storage::cpp2::ExecResponse>::setResult(folly::Executor::KeepAlive<folly::Executor>&&, folly::Try<nebula::storage::cpp2::ExecResponse>&&) (build/bin/nebula-storaged+0x2acab30)
    #17 0x2ac9a3c in folly::futures::detail::Core<nebula::storage::cpp2::ExecResponse>::setResult(folly::Try<nebula::storage::cpp2::ExecResponse>&&) (build/bin/nebula-storaged+0x2ac9a3c)
    #18 0x2ad1b52 in folly::Promise<nebula::storage::cpp2::ExecResponse>::setTry(folly::Try<nebula::storage::cpp2::ExecResponse>&&) (build/bin/nebula-storaged+0x2ad1b52)
    #19 0x2ad0496 in void folly::Promise<nebula::storage::cpp2::ExecResponse>::setValue<nebula::storage::cpp2::ExecResponse>(nebula::storage::cpp2::ExecResponse&&) (build/bin/nebula-storaged+0x2ad0496)
    #20 0x2acf5da in nebula::storage::BaseProcessor<nebula::storage::cpp2::ExecResponse>::onFinished() (build/bin/nebula-storaged+0x2acf5da)
    #21 0x2b17f59 in nebula::storage::BaseProcessor<nebula::storage::cpp2::ExecResponse>::handleAsync(int, int, nebula::cpp2::ErrorCode) (build/bin/nebula-storaged+0x2b17f59)
    #22 0x2b71511 in _ZZN6nebula7storage17AddEdgesProcessor9doProcessERKNS0_4cpp215AddEdgesRequestEENKUlT_E_clINS_4cpp29ErrorCodeEEEDaS6_ (build/bin/nebula-storaged+0x2b71511)
    #23 0x2b886b8 in void folly::detail::function::FunctionTraits<void (nebula::cpp2::ErrorCode)>::callSmall<nebula::storage::AddEdgesProcessor::doProcess(nebula::storage::cpp2::AddEdgesRequest const&)::{lambda(auto:1)#1}>(nebula::cpp2::ErrorCode, folly::detail::function::Data&) (build/bin/nebula-storaged+0x2b886b8)
    #24 0x537e3fb in folly::detail::function::FunctionTraits<void (nebula::cpp2::ErrorCode)>::operator()(nebula::cpp2::ErrorCode) (build/bin/nebula-storaged+0x537e3fb)
    #25 0x536f56e in nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}::operator()(nebula::raftex::AppendLogResult) (build/bin/nebula-storaged+0x536f56e)
    #26 0x537ec21 in auto folly::futures::detail::wrapInvoke<nebula::raftex::AppendLogResult, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>(folly::Try<nebula::raftex::AppendLogResult>&&, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}&&)::{lambda()#1}::operator()() const (build/bin/nebula-storaged+0x537ec21)
    #27 0x538517b in nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}<folly::Unit> folly::futures::detail::InvokeResultWrapper<void>::wrapResult<auto folly::futures::detail::wrapInvoke<nebula::raftex::AppendLogResult, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>(folly::Try<nebula::raftex::AppendLogResult>&&, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}&&)::{lambda()#1}>(folly::Try) (build/bin/nebula-storaged+0x538517b)
    #28 0x537ee68 in auto folly::futures::detail::wrapInvoke<nebula::raftex::AppendLogResult, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>(folly::Try<nebula::raftex::AppendLogResult>&&, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}&&) (build/bin/nebula-storaged+0x537ee68)
    #29 0x537ef65 in folly::Future<folly::futures::detail::valueCallableResult<nebula::raftex::AppendLogResult, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>::value_type> folly::Future<nebula::raftex::AppendLogResult>::thenValue<nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>(folly::futures::detail::valueCallableResult&&) &&::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>&&, folly::Try<nebula::raftex::AppendLogResult>&&)#1}::operator()(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>, folly::Try<nebula::raftex::AppendLogResult>) (build/bin/nebula-storaged+0x537ef65)
    #30 0x538529d in auto folly::futures::detail::CoreCallbackState<folly::Unit, folly::Future<folly::futures::detail::valueCallableResult<nebula::raftex::AppendLogResult, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>::value_type> folly::Future<nebula::raftex::AppendLogResult>::thenValue<nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>(folly::futures::detail::valueCallableResult&&) &&::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>&&, folly::Try<nebula::raftex::AppendLogResult>&&)#1}>::invoke<folly::Executor::KeepAlive, folly::Try>(folly::Executor::KeepAlive&&, folly::Try&&) (build/bin/nebula-storaged+0x538529d)
    #31 0x538535e in decltype(auto) folly::futures::detail::detail_msvc_15_7_workaround::invoke<folly::futures::detail::tryExecutorCallableResult<nebula::raftex::AppendLogResult, folly::Future<folly::futures::detail::valueCallableResult<nebula::raftex::AppendLogResult, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>::value_type> folly::Future<nebula::raftex::AppendLogResult>::thenValue<nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>(folly::futures::detail::valueCallableResult&&) &&::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>&&, folly::Try<nebula::raftex::AppendLogResult>&&)#1}, void>, folly::futures::detail::CoreCallbackState<folly::Unit, folly::Try<nebula::raftex::AppendLogResult>&&>, nebula::raftex::AppendLogResult, 0>(folly::futures::detail::valueCallableResult, folly::futures::detail::CoreCallbackState<folly::Unit, folly::Try<nebula::raftex::AppendLogResult>&&>&, folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>, folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>&&<nebula::raftex::AppendLogResult>&&) (build/bin/nebula-storaged+0x538535e)
    #32 0x5385492 in std::enable_if<!folly::futures::detail::tryExecutorCallableResult<nebula::raftex::AppendLogResult, folly::Try<nebula::raftex::AppendLogResult>&&, void>::ReturnsFuture::value, folly::futures::detail::tryExecutorCallableResult<nebula::raftex::AppendLogResult, folly::Try<nebula::raftex::AppendLogResult>&&, void>::Return>::type folly::futures::detail::FutureBase<nebula::raftex::AppendLogResult>::thenImplementation<folly::Future<folly::futures::detail::valueCallableResult<nebula::raftex::AppendLogResult, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>::value_type> folly::Future<nebula::raftex::AppendLogResult>::thenValue<nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>(folly::futures::detail::valueCallableResult&&) &&::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>&&, folly::Try<nebula::raftex::AppendLogResult>&&)#1}, folly::futures::detail::tryExecutorCallableResult<nebula::raftex::AppendLogResult, folly::Try<nebula::raftex::AppendLogResult>&&, void> >(folly::Future<folly::futures::detail::valueCallableResult<nebula::raftex::AppendLogResult, folly::Future<folly::futures::detail::valueCallableResult<nebula::raftex::AppendLogResult, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>::value_type> folly::Future<nebula::raftex::AppendLogResult>::thenValue<nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>(folly::futures::detail::valueCallableResult&&) &&::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>&&, folly::Try<nebula::raftex::AppendLogResult>&&)#1}>::value_type>, folly::futures::detail::tryExecutorCallableResult<nebula::raftex::AppendLogResult, folly::Try<nebula::raftex::AppendLogResult>&&, void>, folly::futures::detail::InlineContinuation)::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>, folly::Try<nebula::raftex::AppendLogResult>)#1}::operator()(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>, folly::Try<nebula::raftex::AppendLogResult>)::{lambda()#1}::operator()() const (build/bin/nebula-storaged+0x5385492)
    #33 0x538feae in _ZN5folly11makeTryWithIZZNS_7futures6detail10FutureBaseIN6nebula6raftex15AppendLogResultEE18thenImplementationIZNOS_6FutureIS6_E9thenValueIZNS4_7kvstore4Part16asyncAppendBatchEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_8FunctionIFvNS4_4cpp29ErrorCodeEEEEEUlS6_E_EENS9_INS2_19valueCallableResultIS6_T_E10value_typeEEEOSS_EUlONS_8Executor9KeepAliveISX_EEONS_3TryIS6_EEE_NS2_25tryExecutorCallableResultIS6_S14_vEEEENSt9enable_ifIXntsrNT0_13ReturnsFutureE5valueENS18_6ReturnEE4typeESW_S18_NS2_18InlineContinuationEENUlS10_S13_E_clES10_S13_EUlvE_EENS17_IXsrNS_5isTryINS_13invoke_detail6traitsISS_E6resultIEEEE5valueES1L_E4typeESW_ (build/bin/nebula-storaged+0x538feae)
    #34 0x538571b in std::enable_if<!folly::futures::detail::tryExecutorCallableResult<nebula::raftex::AppendLogResult, folly::Try<nebula::raftex::AppendLogResult>&&, void>::ReturnsFuture::value, folly::futures::detail::tryExecutorCallableResult<nebula::raftex::AppendLogResult, folly::Try<nebula::raftex::AppendLogResult>&&, void>::Return>::type folly::futures::detail::FutureBase<nebula::raftex::AppendLogResult>::thenImplementation<folly::Future<folly::futures::detail::valueCallableResult<nebula::raftex::AppendLogResult, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>::value_type> folly::Future<nebula::raftex::AppendLogResult>::thenValue<nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>(folly::futures::detail::valueCallableResult&&) &&::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>&&, folly::Try<nebula::raftex::AppendLogResult>&&)#1}, folly::futures::detail::tryExecutorCallableResult<nebula::raftex::AppendLogResult, folly::Try<nebula::raftex::AppendLogResult>&&, void> >(folly::Future<folly::futures::detail::valueCallableResult<nebula::raftex::AppendLogResult, folly::Future<folly::futures::detail::valueCallableResult<nebula::raftex::AppendLogResult, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>::value_type> folly::Future<nebula::raftex::AppendLogResult>::thenValue<nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>(folly::futures::detail::valueCallableResult&&) &&::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>&&, folly::Try<nebula::raftex::AppendLogResult>&&)#1}>::value_type>, folly::futures::detail::tryExecutorCallableResult<nebula::raftex::AppendLogResult, folly::Try<nebula::raftex::AppendLogResult>&&, void>, folly::futures::detail::InlineContinuation)::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>, folly::Try<nebula::raftex::AppendLogResult>)#1}::operator()(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>, folly::Try<nebula::raftex::AppendLogResult>) (build/bin/nebula-storaged+0x538571b)
    #35 0x5397e56 in _ZZN5folly7futures6detail4CoreIN6nebula6raftex15AppendLogResultEE11setCallbackIZNS1_10FutureBaseIS5_E18thenImplementationIZNOS_6FutureIS5_E9thenValueIZNS3_7kvstore4Part16asyncAppendBatchEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_8FunctionIFvNS3_4cpp29ErrorCodeEEEEEUlS5_E_EENSB_INS1_19valueCallableResultIS5_T_E10value_typeEEEOSU_EUlONS_8Executor9KeepAliveISZ_EEONS_3TryIS5_EEE_NS1_25tryExecutorCallableResultIS5_S16_vEEEENSt9enable_ifIXntsrNT0_13ReturnsFutureE5valueENS1A_6ReturnEE4typeESY_S1A_NS1_18InlineContinuationEEUlS12_S15_E_EEvSY_OSt10shared_ptrINS_14RequestContextEES1F_ENUlRNS1_8CoreBaseES12_PNS_17exception_wrapperEE_clES1M_S12_S1O_ (build/bin/nebula-storaged+0x5397e56)
    #36 0x539f4cd in _ZN5folly6detail8function14FunctionTraitsIFvRNS_7futures6detail8CoreBaseEONS_8Executor9KeepAliveIS7_EEPNS_17exception_wrapperEEE7callBigIZNS4_4CoreIN6nebula6raftex15AppendLogResultEE11setCallbackIZNS4_10FutureBaseISJ_E18thenImplementationIZNOS_6FutureISJ_E9thenValueIZNSH_7kvstore4Part16asyncAppendBatchEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_8FunctionIFvNSH_4cpp29ErrorCodeEEEEEUlSJ_E_EENSP_INS4_19valueCallableResultISJ_T_E10value_typeEEEOS18_EUlSA_ONS_3TryISJ_EEE_NS4_25tryExecutorCallableResultISJ_S1G_vEEEENSt9enable_ifIXntsrNT0_13ReturnsFutureE5valueENS1K_6ReturnEE4typeES1C_S1K_NS4_18InlineContinuationEEUlSA_S1F_E_EEvS1C_OSt10shared_ptrINS_14RequestContextEES1P_EUlS6_SA_SC_E_EEvS6_SA_SC_RNS1_4DataE (build/bin/nebula-storaged+0x539f4cd)
    #37 0x5e6ea97 in folly::futures::detail::CoreBase::doCallback(folly::Executor::KeepAlive<folly::Executor>&&, folly::futures::detail::State) (build/bin/nebula-storaged+0x5e6ea97)
    #38 0x5e6f21e in folly::futures::detail::CoreBase::setCallback_(folly::Function<void (folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<folly::Executor>&&, folly::exception_wrapper*)>&&, std::shared_ptr<folly::RequestContext>&&, folly::futures::detail::InlineContinuation) (build/bin/nebula-storaged+0x5e6f21e)
    #39 0x5398046 in _ZN5folly7futures6detail4CoreIN6nebula6raftex15AppendLogResultEE11setCallbackIZNS1_10FutureBaseIS5_E18thenImplementationIZNOS_6FutureIS5_E9thenValueIZNS3_7kvstore4Part16asyncAppendBatchEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_8FunctionIFvNS3_4cpp29ErrorCodeEEEEEUlS5_E_EENSB_INS1_19valueCallableResultIS5_T_E10value_typeEEEOSU_EUlONS_8Executor9KeepAliveISZ_EEONS_3TryIS5_EEE_NS1_25tryExecutorCallableResultIS5_S16_vEEEENSt9enable_ifIXntsrNT0_13ReturnsFutureE5valueENS1A_6ReturnEE4typeESY_S1A_NS1_18InlineContinuationEEUlS12_S15_E_EEvSY_OSt10shared_ptrINS_14RequestContextEES1F_ (build/bin/nebula-storaged+0x5398046)
    #40 0x539035e in _ZN5folly7futures6detail10FutureBaseIN6nebula6raftex15AppendLogResultEE12setCallback_IZNS6_18thenImplementationIZNOS_6FutureIS5_E9thenValueIZNS3_7kvstore4Part16asyncAppendBatchEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_8FunctionIFvNS3_4cpp29ErrorCodeEEEEEUlS5_E_EENS9_INS1_19valueCallableResultIS5_T_E10value_typeEEEOSS_EUlONS_8Executor9KeepAliveISX_EEONS_3TryIS5_EEE_NS1_25tryExecutorCallableResultIS5_S14_vEEEENSt9enable_ifIXntsrNT0_13ReturnsFutureE5valueENS18_6ReturnEE4typeESW_S18_NS1_18InlineContinuationEEUlS10_S13_E_EEvSW_S1D_ (build/bin/nebula-storaged+0x539035e)
    #41 0x5385bfe in std::enable_if<!folly::futures::detail::tryExecutorCallableResult<nebula::raftex::AppendLogResult, folly::Try<nebula::raftex::AppendLogResult>&&, void>::ReturnsFuture::value, folly::futures::detail::tryExecutorCallableResult<nebula::raftex::AppendLogResult, folly::Try<nebula::raftex::AppendLogResult>&&, void>::Return>::type folly::futures::detail::FutureBase<nebula::raftex::AppendLogResult>::thenImplementation<folly::Future<folly::futures::detail::valueCallableResult<nebula::raftex::AppendLogResult, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>::value_type> folly::Future<nebula::raftex::AppendLogResult>::thenValue<nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>(folly::futures::detail::valueCallableResult&&) &&::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>&&, folly::Try<nebula::raftex::AppendLogResult>&&)#1}, folly::futures::detail::tryExecutorCallableResult<nebula::raftex::AppendLogResult, folly::Try<nebula::raftex::AppendLogResult>&&, void> >(folly::Future<folly::futures::detail::valueCallableResult<nebula::raftex::AppendLogResult, folly::Future<folly::futures::detail::valueCallableResult<nebula::raftex::AppendLogResult, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>::value_type> folly::Future<nebula::raftex::AppendLogResult>::thenValue<nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>(folly::futures::detail::valueCallableResult&&) &&::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult&&>&&, folly::Try<nebula::raftex::AppendLogResult>&&)#1}>::value_type>, folly::futures::detail::tryExecutorCallableResult<nebula::raftex::AppendLogResult, folly::Try<nebula::raftex::AppendLogResult>&&, void>, folly::futures::detail::InlineContinuation) (build/bin/nebula-storaged+0x5385bfe)
    #42 0x537f129 in folly::Future<folly::futures::detail::valueCallableResult<nebula::raftex::AppendLogResult, nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>::value_type> folly::Future<nebula::raftex::AppendLogResult>::thenValue<nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>)::{lambda(nebula::raftex::AppendLogResult)#1}>(folly::futures::detail::valueCallableResult&&) && (build/bin/nebula-storaged+0x537f129)
    #43 0x536f771 in nebula::kvstore::Part::asyncAppendBatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>) (build/bin/nebula-storaged+0x536f771)
    #44 0x540219a in nebula::kvstore::NebulaStore::asyncAppendBatch(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, folly::Function<void (nebula::cpp2::ErrorCode)>) (build/bin/nebula-storaged+0x540219a)
    #45 0x2b74b01 in nebula::storage::AddEdgesProcessor::doProcess(nebula::storage::cpp2::AddEdgesRequest const&) (build/bin/nebula-storaged+0x2b74b01)
    #46 0x2b70ecc in nebula::storage::AddEdgesProcessor::process(nebula::storage::cpp2::AddEdgesRequest const&) (build/bin/nebula-storaged+0x2b70ecc)
    #47 0x2ee1520 in nebula::storage::ChainAddEdgesProcessorLocal::forwardToDelegateProcessor() (build/bin/nebula-storaged+0x2ee1520)
    #48 0x2efaf10 in nebula::storage::ResumeAddEdgeProcessor::processLocal(nebula::cpp2::ErrorCode) (build/bin/nebula-storaged+0x2efaf10)
    #49 0x2e9dd57 in _ZZZN6nebula7storage18TransactionManager12addChainTaskEPNS0_18ChainBaseProcessorEENKUlvE_clEvENKUlOT_E0_clINS_4cpp29ErrorCodeEEEDaS6_ (build/bin/nebula-storaged+0x2e9dd57)
    #50 0x2ea5c21 in auto folly::futures::detail::wrapInvoke<nebula::cpp2::ErrorCode, nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}>(folly::Try<{lambda()#1}>&&, nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}&&)::{lambda()#1}::operator()() const (build/bin/nebula-storaged+0x2ea5c21)
    #51 0x2ea9918 in folly::SemiFuture<nebula::cpp2::ErrorCode> folly::futures::detail::InvokeResultWrapperBase<folly::SemiFuture<nebula::cpp2::ErrorCode> >::wrapResult<auto folly::futures::detail::wrapInvoke<nebula::cpp2::ErrorCode, nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}>(folly::Try<{lambda()#1}>&&, nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}&&)::{lambda()#1}>({lambda()#1}) (build/bin/nebula-storaged+0x2ea9918)
    #52 0x2ea5ddc in auto folly::futures::detail::wrapInvoke<nebula::cpp2::ErrorCode, nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}>(folly::Try<{lambda()#1}>&&, nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}&&) (build/bin/nebula-storaged+0x2ea5ddc)
    #53 0x2ea5b72 in folly::Future<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type> folly::Future<nebula::cpp2::ErrorCode>::thenValue<nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}>(nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}) &&::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type>&&, folly::Try<nebula::cpp2::ErrorCode>&&)#1}::operator()(folly::Executor::KeepAlive, folly::Try) (build/bin/nebula-storaged+0x2ea5b72)
    #54 0x2eaa1b8 in auto folly::futures::detail::CoreCallbackState<nebula::cpp2::ErrorCode, folly::Future<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type> folly::Future<nebula::cpp2::ErrorCode>::thenValue<nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}>(nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}) &&::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type>&&, folly::Try<nebula::cpp2::ErrorCode>&&)#1}>::invoke<folly::Executor, folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type>&&>(folly::Executor&&, folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type>&&) (build/bin/nebula-storaged+0x2eaa1b8)
    #55 0x2eaa133 in auto folly::futures::detail::CoreCallbackState<nebula::cpp2::ErrorCode, folly::Future<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type> folly::Future<nebula::cpp2::ErrorCode>::thenValue<nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}>(nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}) &&::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type>&&, folly::Try<nebula::cpp2::ErrorCode>&&)#1}>::tryInvoke<folly::Executor, folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type>&&>(folly::Executor&&, folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type>&&)::{lambda()#1}::operator()() const (build/bin/nebula-storaged+0x2eaa133)
    #56 0x2eb2bf8 in _ZN5folly19makeTryWithNoUnwrapIZNS_7futures6detail17CoreCallbackStateIN6nebula4cpp29ErrorCodeEZNOS_6FutureIS6_E9thenValueIZZNS4_7storage18TransactionManager12addChainTaskEPNSA_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENS7_INS2_19valueCallableResultIS6_SF_E10value_typeEEESG_EUlONS_8Executor9KeepAliveISM_EEONS_3TryIS6_EEE_E9tryInvokeIJSO_SR_EEEDaDpOT_EUlvE_EENSt9enable_ifIXntsrSt7is_sameINS_13invoke_detail6traitsISF_E6resultIEEvE5valueENSQ_IS16_EEE4typeESG_ (build/bin/nebula-storaged+0x2eb2bf8)
    #57 0x2eae2d2 in _ZN5folly11makeTryWithIZNS_7futures6detail17CoreCallbackStateIN6nebula4cpp29ErrorCodeEZNOS_6FutureIS6_E9thenValueIZZNS4_7storage18TransactionManager12addChainTaskEPNSA_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENS7_INS2_19valueCallableResultIS6_SF_E10value_typeEEESG_EUlONS_8Executor9KeepAliveISM_EEONS_3TryIS6_EEE_E9tryInvokeIJSO_SR_EEEDaDpOT_EUlvE_EENSt9enable_ifIXntsrNS_5isTryINS_13invoke_detail6traitsISF_E6resultIEEEE5valueENSQ_IS16_EEE4typeESG_ (build/bin/nebula-storaged+0x2eae2d2)
    #58 0x2eaa360 in auto folly::futures::detail::CoreCallbackState<nebula::cpp2::ErrorCode, folly::Future<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type> folly::Future<nebula::cpp2::ErrorCode>::thenValue<nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}>(nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}) &&::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type>&&, folly::Try<nebula::cpp2::ErrorCode>&&)#1}>::tryInvoke<folly::Executor, folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type>&&>(folly::Executor&&, folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type>&&) (build/bin/nebula-storaged+0x2eaa360)
    #59 0x2eaa474 in decltype(auto) folly::futures::detail::detail_msvc_15_7_workaround::tryInvoke<folly::futures::detail::tryExecutorCallableResult<nebula::cpp2::ErrorCode, folly::Future<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type> folly::Future<nebula::cpp2::ErrorCode>::thenValue<nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}>(nebula::storage::TransactionManager::addChainTask(nebula::storage::ChainBaseProcessor*)::{lambda()#1}::operator()() const::{lambda(auto:1&&)#2}) &&::{lambda(folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type>&&, folly::Try<nebula::cpp2::ErrorCode>&&)#1}, void>, folly::futures::detail::CoreCallbackState<nebula::cpp2::ErrorCode, folly::Try<nebula::cpp2::ErrorCode> >, nebula::cpp2::ErrorCode, 0>({lambda()#1}, folly::futures::detail::CoreCallbackState<nebula::cpp2::ErrorCode, folly::Try<nebula::cpp2::ErrorCode> >&, folly::Executor::KeepAlive, folly::Executor::KeepAlive<folly::futures::detail::valueCallableResult<nebula::cpp2::ErrorCode, {lambda()#1}>::value_type><nebula::cpp2::ErrorCode>&&) (build/bin/nebula-storaged+0x2eaa474)
    #60 0x2ea9bc8 in _ZZN5folly7futures6detail10FutureBaseIN6nebula4cpp29ErrorCodeEE18thenImplementationIZNOS_6FutureIS5_E9thenValueIZZNS3_7storage18TransactionManager12addChainTaskEPNSB_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENS8_INS1_19valueCallableResultIS5_SG_E10value_typeEEESH_EUlONS_8Executor9KeepAliveISN_EEONS_3TryIS5_EEE_NS1_25tryExecutorCallableResultIS5_SU_vEEEENSt9enable_ifIXsrNT0_13ReturnsFutureE5valueENSY_6ReturnEE4typeESH_SY_NS1_18InlineContinuationEENUlSQ_ST_E_clESQ_ST_ (build/bin/nebula-storaged+0x2ea9bc8)
    #61 0x2eb2ffc in _ZZN5folly7futures6detail4CoreIN6nebula4cpp29ErrorCodeEE11setCallbackIZNS1_10FutureBaseIS5_E18thenImplementationIZNOS_6FutureIS5_E9thenValueIZZNS3_7storage18TransactionManager12addChainTaskEPNSE_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENSB_INS1_19valueCallableResultIS5_SJ_E10value_typeEEESK_EUlONS_8Executor9KeepAliveISQ_EEONS_3TryIS5_EEE_NS1_25tryExecutorCallableResultIS5_SX_vEEEENSt9enable_ifIXsrNT0_13ReturnsFutureE5valueENS11_6ReturnEE4typeESK_S11_NS1_18InlineContinuationEEUlST_SW_E_EEvSK_OSt10shared_ptrINS_14RequestContextEES16_ENUlRNS1_8CoreBaseEST_PNS_17exception_wrapperEE_clES1D_ST_S1F_ (build/bin/nebula-storaged+0x2eb2ffc)
    #62 0x2ebba51 in _ZN5folly6detail8function14FunctionTraitsIFvRNS_7futures6detail8CoreBaseEONS_8Executor9KeepAliveIS7_EEPNS_17exception_wrapperEEE9callSmallIZNS4_4CoreIN6nebula4cpp29ErrorCodeEE11setCallbackIZNS4_10FutureBaseISJ_E18thenImplementationIZNOS_6FutureISJ_E9thenValueIZZNSH_7storage18TransactionManager12addChainTaskEPNSS_18ChainBaseProcessorEENKUlvE_clEvEUlOT_E0_EENSP_INS4_19valueCallableResultISJ_SX_E10value_typeEEESY_EUlSA_ONS_3TryISJ_EEE_NS4_25tryExecutorCallableResultISJ_S17_vEEEENSt9enable_ifIXsrNT0_13ReturnsFutureE5valueENS1B_6ReturnEE4typeESY_S1B_NS4_18InlineContinuationEEUlSA_S16_E_EEvSY_OSt10shared_ptrINS_14RequestContextEES1G_EUlS6_SA_SC_E_EEvS6_SA_SC_RNS1_4DataE (build/bin/nebula-storaged+0x2ebba51)
    #63 0x5e6dbbb  (build/bin/nebula-storaged+0x5e6dbbb)
    #64 0x5e49e05 in folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) (build/bin/nebula-storaged+0x5e49e05)
    #65 0x5e3cecb  (build/bin/nebula-storaged+0x5e3cecb)
    #66 0x5eb5fba in bool folly::AtomicNotificationQueue<folly::Function<void ()> >::drive<folly::EventBase::FuncRunner&>(folly::EventBase::FuncRunner&) (build/bin/nebula-storaged+0x5eb5fba)
    #67 0x5eb712c in non-virtual thunk to folly::EventBaseAtomicNotificationQueue<folly::Function<void ()>, folly::EventBase::FuncRunner>::handlerReady(unsigned short) (build/bin/nebula-storaged+0x5eb712c)
    #68 0x5f8780e  (build/bin/nebula-storaged+0x5f8780e)
    #69 0x5f8812e in event_base_loop (build/bin/nebula-storaged+0x5f8812e)
    #70 0x5eb1e7d in folly::EventBase::loopBody(int, bool) (build/bin/nebula-storaged+0x5eb1e7d)
    #71 0x5eb2325 in folly::EventBase::loop() (build/bin/nebula-storaged+0x5eb2325)
    #72 0x5eb3dd5 in folly::EventBase::loopForever() (build/bin/nebula-storaged+0x5eb3dd5)
    #73 0x5e3da08 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) (build/bin/nebula-storaged+0x5e3da08)
    #74 0x5e4b5d8 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) (build/bin/nebula-storaged+0x5e4b5d8)
    #75 0x29f424d in folly::detail::function::FunctionTraits<void ()>::operator()() (build/bin/nebula-storaged+0x29f424d)
    #76 0x2a16cd1 in folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}::operator()() (build/bin/nebula-storaged+0x2a16cd1)
    #77 0x2a3b582 in void std::__invoke_impl<void, folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(std::__invoke_other, folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}&&) (build/bin/nebula-storaged+0x2a3b582)
    #78 0x2a2f3fb in std::__invoke_result<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>::type std::__invoke<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(std::__invoke_result&&, (folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}&&)...) (build/bin/nebula-storaged+0x2a2f3fb)
    #79 0x2ab2ba3 in decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (build/bin/nebula-storaged+0x2ab2ba3)
    #80 0x2ab1342 in std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}> >::operator()() (build/bin/nebula-storaged+0x2ab1342)
    #81 0x2aab369 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}> > >::_M_run() (build/bin/nebula-storaged+0x2aab369)
    #82 0x62abf7e in execute_native_thread_routine (build/bin/nebula-storaged+0x62abf7e)
    #83 0x7ffff729eea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
    #84 0x7ffff6fc79fc in __clone (/lib64/libc.so.6+0xfe9fc)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (build/bin/nebula-storaged+0x2ee9c9a) in nebula::MemoryLockGuard<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::forceUnlock()
Thread T143 (IOThreadPool0) created by T30 (executor-pri3-2) here:
    #0 0x2909800 in pthread_create (build/bin/nebula-storaged+0x2909800)
    #1 0x62ac074 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (build/bin/nebula-storaged+0x62ac074)

Thread T30 (executor-pri3-2) created by T0 here:
    #0 0x2909800 in pthread_create (build/bin/nebula-storaged+0x2909800)
    #1 0x5c014d9 in apache::thrift::concurrency::PthreadThread::start() (build/bin/nebula-storaged+0x5c014d9)
    #2 0x5bf200b in apache::thrift::concurrency::ThreadManager::Impl::addWorker(unsigned long) (build/bin/nebula-storaged+0x5bf200b)
    #3 0x5bf8f2c in apache::thrift::concurrency::PriorityThreadManager::PriorityImpl::start() (build/bin/nebula-storaged+0x5bf8f2c)
    #4 0x2a2bbe3 in nebula::storage::StorageServer::start() (build/bin/nebula-storaged+0x2a2bbe3)
    #5 0x29f2333 in main (build/bin/nebula-storaged+0x29f2333)
    #6 0x7ffff6eeb554 in __libc_start_main (/lib64/libc.so.6+0x22554)

==85851==ABORTING

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

Provide logs and configs, or any other context to trace the problem.
@kikimo kikimo added the type/bug Type: something is unexpected label Sep 17, 2021
@kikimo kikimo added this to the v2.6.0 milestone Sep 17, 2021
@jamieliu1023 jamieliu1023 mentioned this issue Sep 18, 2021
Closed
@liuyu85cn
Copy link
Contributor

caused by two thread not sync correctly. Fixed in pr #2525

@jamieliu1023 jamieliu1023 mentioned this issue Oct 2, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liuyu85cn liuyu85cn

Labels
type/bug Type: something is unexpected
Projects
None yet
Milestone
v2.6.0
Development

No branches or pull requests
2 participants
@kikimo @liuyu85cn