Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ROOT6, ROOT628] Fatal system signal has occurred during exit #40347

Open
makortel opened this issue Dec 16, 2022 · 42 comments
Open

[ROOT6, ROOT628] Fatal system signal has occurred during exit #40347

makortel opened this issue Dec 16, 2022 · 42 comments
Labels
core-pending pending-signatures

Comments

@makortel
Copy link
Contributor

We are seeing occasional, random job failures because of

Fatal system signal has occurred during exit

in ROOT6 IBs, at least since CMSSW_13_0_ROOT6_X_2022-12-09-2300 (likely earlier). I'm staring to collect now at least how frequent the job failure is

  • CMSSW_13_0_ROOT6_X_2022-12-09-2300: 138.4 step 2
  • CMSSW_13_0_ROOT6_X_2022-12-12-2300: 4.24 step 3
  • CMSSW_13_0_ROOT6_X_2022-12-14-2300: 136.877 step 3
  • CMSSW_13_0_ROOT6_X_2022-12-15-2300: 4.44 step 3 and 4.54 step 3
@makortel
Copy link
Contributor Author

assign core

@cmsbuild
Copy link
Contributor

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

@cmsbuild
Copy link
Contributor

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@Dr15Jones
Copy link
Contributor

I've been running WF 4.44 in that IB using 4 threads and so far (after 3 runs) I have yet to be able to reproduce the failure. Probably a threading issue.

@Dr15Jones
Copy link
Contributor

So I ran valgrind and got the following four errors at the end of the job

==2645757== Invalid free() / delete / delete[] / realloc()
==2645757==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2645757==    by 0x1551C81F: ???
==2645757==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2645757==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2645757==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2645757==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2645757==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2645757==  Address 0x712de110 is 0 bytes inside a block of size 18 free'd
==2645757==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2645757==    by 0x1551C81F: ???
==2645757==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2645757==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2645757==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2645757==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2645757==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2645757==  Block was alloc'd at
==2645757==    at 0x403AF01: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2645757==    by 0x512ECA2: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (basic_string.tcc:219)
==2645757==    by 0x512DD9B: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) (basic_string.h:539)
==2645757==    by 0x1551C228: ???
==2645757==    by 0x1551C8AF: ???
==2645757==    by 0x1551C94C: ???
==2645757==    by 0x1551C93F: ???
==2645757==    by 0x2085C661: (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F2F3728: cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F27ED27: cling::Interpreter::executeTransaction(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F302489: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F305248: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.s

==2645757== Invalid free() / delete / delete[] / realloc()
==2645757==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2645757==    by 0x1551C841: ???
==2645757==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2645757==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2645757==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2645757==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2645757==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2645757==  Address 0x71284050 is 0 bytes inside a block of size 20 free'd
==2645757==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2645757==    by 0x1551C841: ???
==2645757==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2645757==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2645757==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2645757==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2645757==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2645757==  Block was alloc'd at
==2645757==    at 0x403AF01: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2645757==    by 0x512ECA2: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (basic_string.tcc:219)
==2645757==    by 0x512DD9B: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) (basic_string.h:539)
==2645757==    by 0x1551C1B4: ???
==2645757==    by 0x1551C8AF: ???
==2645757==    by 0x1551C94C: ???
==2645757==    by 0x1551C93F: ???
==2645757==    by 0x2085C661: (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F2F3728: cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F27ED27: cling::Interpreter::executeTransaction(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F302489: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F305248: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.s

==2645757== Invalid free() / delete / delete[] / realloc()
==2645757==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2645757==    by 0x1551C863: ???
==2645757==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2645757==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2645757==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2645757==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2645757==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2645757==  Address 0x7127fc50 is 0 bytes inside a block of size 18 free'd
==2645757==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2645757==    by 0x1551C863: ???
==2645757==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2645757==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2645757==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2645757==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2645757==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2645757==  Block was alloc'd at
==2645757==    at 0x403AF01: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2645757==    by 0x512ECA2: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (basic_string.tcc:219)
==2645757==    by 0x512DD9B: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) (basic_string.h:539)
==2645757==    by 0x1551C140: ???
==2645757==    by 0x1551C8AF: ???
==2645757==    by 0x1551C94C: ???
==2645757==    by 0x1551C93F: ???
==2645757==    by 0x2085C661: (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F2F3728: cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F27ED27: cling::Interpreter::executeTransaction(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F302489: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F305248: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.s

==2645757== Invalid free() / delete / delete[] / realloc()
==2645757==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2645757==    by 0x1551C871: ???
==2645757==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2645757==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2645757==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2645757==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2645757==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2645757==  Address 0x7127fb30 is 0 bytes inside a block of size 26 free'd
==2645757==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2645757==    by 0x1551C871: ???
==2645757==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2645757==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2645757==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2645757==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2645757==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2645757==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2645757==  Block was alloc'd at
==2645757==    at 0x403AF01: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2645757==    by 0x512ECA2: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (basic_string.tcc:219)
==2645757==    by 0x512DD9B: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) (basic_string.h:539)
==2645757==    by 0x1551C106: ???
==2645757==    by 0x1551C8AF: ???
==2645757==    by 0x1551C94C: ???
==2645757==    by 0x1551C93F: ???
==2645757==    by 0x2085C661: (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F2F3728: cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F27ED27: cling::Interpreter::executeTransaction(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F302489: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2645757==    by 0x1F305248: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.s

@pcanal FYI

@Dr15Jones
Copy link
Contributor

So after a couple more attempts (since the other tries didn't show the failure) I was able to get a failure with a longer stack trace

==2792291== Invalid free() / delete / delete[] / realloc()
==2792291==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x6E72480E: ???
==2792291==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2792291==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2792291==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2792291==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2792291==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2792291==  Address 0x1b66bac0 is 0 bytes inside a block of size 29 free'd
==2792291==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x6E72480E: ???
==2792291==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2792291==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2792291==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2792291==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2792291==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2792291==  Block was alloc'd at
==2792291==    at 0x403AF01: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x512ECA2: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (basic_string.tcc:219)
==2792291==    by 0x512DD9B: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) (basic_string.h:539)
==2792291==    by 0x6E724262: ???
==2792291==    by 0x6E7248AF: ???
==2792291==    by 0x6E72494C: ???
==2792291==    by 0x6E72493F: ???
==2792291==    by 0x2085C661: (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F2F3728: cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F27ED27: cling::Interpreter::executeTransaction(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F302489: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F305248: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F27AF9A: cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F05BD7A: ExecAutoParse(char const*, bool, cling::Interpreter*) (TCling.cxx:6305)
==2792291==    by 0x1F05C602: TCling::AutoParseImplRecurse(char const*, bool) (TCling.cxx:6410)
==2792291==    by 0x1F05C951: TCling::AutoParseImplRecurse(char const*, bool) (TCling.cxx:6446)
==2792291==    by 0x1F05CC82: TCling::AutoParse(char const*) (TCling.cxx:6495)
==2792291==    by 0x1F044564: TClingLookupHelper__AutoParse(char const*) (TCling.cxx:901)
==2792291==    by 0x1EE9F324: ROOT::TMetaUtils::TClingLookupHelper::GetPartiallyDesugaredNameWithScopeHandling(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) (TClingUtils.cxx:581)
==2792291==    by 0x619B73D: TClassEdit::TSplitType::ShortType(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, int) (TClassEdit.cxx:454)
==2792291==    by 0x619CDD6: TClassEdit::GetNormalizedName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::basic_string_view<char, std::char_traits<char> >) (TClassEdit.cxx:871)
==2792291==    by 0x61BF522: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (TClass.cxx:3056)
==2792291==    by 0x61BF1AC: TClass::GetClass(char const*, bool, bool) (TClass.cxx:2971)
==2792291==    by 0x4207098: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==2792291==    by 0x4205C84: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==2792291==    by 0x4F2BD35: edm::BranchDescription::initFromDictionary() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==2792291==    by 0x4F2C1A0: edm::BranchDescription::BranchDescription(edm::BranchType const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::Hash<1> const&, edm::TypeWithDict const&, bool, bool, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==2792291==    by 0x4C7B678: edm::ProductRegistryHelper::addToRegistry(__gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, __gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, edm::ModuleDescription const&, edm::ProductRegistry&, edm::ProductRegistryHelper*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==2792291==    by 0x4C77FEC: edm::ProducerBase::registerProducts(edm::ProducerBase*, edm::ProductRegistry*, edm::ModuleDescription const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==2792291==    by 0x41463ABF: edm::maker::ModuleHolderT<edm::stream::EDProducerAdaptorBase>::registerProductsAndCallbacks(edm::ProductRegistry*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/pluginRecoBTagCombinedPlugins.so)

==2792291== Invalid free() / delete / delete[] / realloc()
==2792291==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x6E72481F: ???
==2792291==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2792291==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2792291==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2792291==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2792291==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2792291==  Address 0x57b4a4f0 is 0 bytes inside a block of size 18 free'd
==2792291==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x6E72481F: ???
==2792291==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2792291==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2792291==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2792291==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2792291==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2792291==  Block was alloc'd at
==2792291==    at 0x403AF01: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x512ECA2: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (basic_string.tcc:219)
==2792291==    by 0x512DD9B: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) (basic_string.h:539)
==2792291==    by 0x6E724228: ???
==2792291==    by 0x6E7248AF: ???
==2792291==    by 0x6E72494C: ???
==2792291==    by 0x6E72493F: ???
==2792291==    by 0x2085C661: (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F2F3728: cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F27ED27: cling::Interpreter::executeTransaction(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F302489: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F305248: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F27AF9A: cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F05BD7A: ExecAutoParse(char const*, bool, cling::Interpreter*) (TCling.cxx:6305)
==2792291==    by 0x1F05C602: TCling::AutoParseImplRecurse(char const*, bool) (TCling.cxx:6410)
==2792291==    by 0x1F05C951: TCling::AutoParseImplRecurse(char const*, bool) (TCling.cxx:6446)
==2792291==    by 0x1F05CC82: TCling::AutoParse(char const*) (TCling.cxx:6495)
==2792291==    by 0x1F044564: TClingLookupHelper__AutoParse(char const*) (TCling.cxx:901)
==2792291==    by 0x1EE9F324: ROOT::TMetaUtils::TClingLookupHelper::GetPartiallyDesugaredNameWithScopeHandling(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) (TClingUtils.cxx:581)
==2792291==    by 0x619B73D: TClassEdit::TSplitType::ShortType(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, int) (TClassEdit.cxx:454)
==2792291==    by 0x619CDD6: TClassEdit::GetNormalizedName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::basic_string_view<char, std::char_traits<char> >) (TClassEdit.cxx:871)
==2792291==    by 0x61BF522: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (TClass.cxx:3056)
==2792291==    by 0x61BF1AC: TClass::GetClass(char const*, bool, bool) (TClass.cxx:2971)
==2792291==    by 0x4207098: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==2792291==    by 0x4205C84: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==2792291==    by 0x4F2BD35: edm::BranchDescription::initFromDictionary() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==2792291==    by 0x4F2C1A0: edm::BranchDescription::BranchDescription(edm::BranchType const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::Hash<1> const&, edm::TypeWithDict const&, bool, bool, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==2792291==    by 0x4C7B678: edm::ProductRegistryHelper::addToRegistry(__gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, __gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, edm::ModuleDescription const&, edm::ProductRegistry&, edm::ProductRegistryHelper*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==2792291==    by 0x4C77FEC: edm::ProducerBase::registerProducts(edm::ProducerBase*, edm::ProductRegistry*, edm::ModuleDescription const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==2792291==    by 0x41463ABF: edm::maker::ModuleHolderT<edm::stream::EDProducerAdaptorBase>::registerProductsAndCallbacks(edm::ProductRegistry*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/pluginRecoBTagCombinedPlugins.so)

==2792291== Invalid free() / delete / delete[] / realloc()
==2792291==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x6E724841: ???
==2792291==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2792291==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2792291==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2792291==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2792291==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2792291==  Address 0x66ae9eb0 is 0 bytes inside a block of size 20 free'd
==2792291==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x6E724841: ???
==2792291==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2792291==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2792291==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2792291==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2792291==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2792291==  Block was alloc'd at
==2792291==    at 0x403AF01: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x512ECA2: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (basic_string.tcc:219)
==2792291==    by 0x512DD9B: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) (basic_string.h:539)
==2792291==    by 0x6E7241B4: ???
==2792291==    by 0x6E7248AF: ???
==2792291==    by 0x6E72494C: ???
==2792291==    by 0x6E72493F: ???
==2792291==    by 0x2085C661: (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F2F3728: cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F27ED27: cling::Interpreter::executeTransaction(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F302489: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F305248: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F27AF9A: cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F05BD7A: ExecAutoParse(char const*, bool, cling::Interpreter*) (TCling.cxx:6305)
==2792291==    by 0x1F05C602: TCling::AutoParseImplRecurse(char const*, bool) (TCling.cxx:6410)
==2792291==    by 0x1F05C951: TCling::AutoParseImplRecurse(char const*, bool) (TCling.cxx:6446)
==2792291==    by 0x1F05CC82: TCling::AutoParse(char const*) (TCling.cxx:6495)
==2792291==    by 0x1F044564: TClingLookupHelper__AutoParse(char const*) (TCling.cxx:901)
==2792291==    by 0x1EE9F324: ROOT::TMetaUtils::TClingLookupHelper::GetPartiallyDesugaredNameWithScopeHandling(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) (TClingUtils.cxx:581)
==2792291==    by 0x619B73D: TClassEdit::TSplitType::ShortType(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, int) (TClassEdit.cxx:454)
==2792291==    by 0x619CDD6: TClassEdit::GetNormalizedName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::basic_string_view<char, std::char_traits<char> >) (TClassEdit.cxx:871)
==2792291==    by 0x61BF522: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (TClass.cxx:3056)
==2792291==    by 0x61BF1AC: TClass::GetClass(char const*, bool, bool) (TClass.cxx:2971)
==2792291==    by 0x4207098: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==2792291==    by 0x4205C84: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==2792291==    by 0x4F2BD35: edm::BranchDescription::initFromDictionary() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==2792291==    by 0x4F2C1A0: edm::BranchDescription::BranchDescription(edm::BranchType const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::Hash<1> const&, edm::TypeWithDict const&, bool, bool, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==2792291==    by 0x4C7B678: edm::ProductRegistryHelper::addToRegistry(__gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, __gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, edm::ModuleDescription const&, edm::ProductRegistry&, edm::ProductRegistryHelper*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==2792291==    by 0x4C77FEC: edm::ProducerBase::registerProducts(edm::ProducerBase*, edm::ProductRegistry*, edm::ModuleDescription const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==2792291==    by 0x41463ABF: edm::maker::ModuleHolderT<edm::stream::EDProducerAdaptorBase>::registerProductsAndCallbacks(edm::ProductRegistry*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/pluginRecoBTagCombinedPlugins.so)

==2792291== Invalid free() / delete / delete[] / realloc()
==2792291==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x6E724863: ???
==2792291==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2792291==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2792291==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2792291==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2792291==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2792291==  Address 0x62fc46c0 is 0 bytes inside a block of size 18 free'd
==2792291==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x6E724863: ???
==2792291==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2792291==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2792291==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2792291==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2792291==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2792291==  Block was alloc'd at
==2792291==    at 0x403AF01: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x512ECA2: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (basic_string.tcc:219)
==2792291==    by 0x512DD9B: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) (basic_string.h:539)
==2792291==    by 0x6E724140: ???
==2792291==    by 0x6E7248AF: ???
==2792291==    by 0x6E72494C: ???
==2792291==    by 0x6E72493F: ???
==2792291==    by 0x2085C661: (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F2F3728: cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F27ED27: cling::Interpreter::executeTransaction(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F302489: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F305248: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F27AF9A: cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F05BD7A: ExecAutoParse(char const*, bool, cling::Interpreter*) (TCling.cxx:6305)
==2792291==    by 0x1F05C602: TCling::AutoParseImplRecurse(char const*, bool) (TCling.cxx:6410)
==2792291==    by 0x1F05C951: TCling::AutoParseImplRecurse(char const*, bool) (TCling.cxx:6446)
==2792291==    by 0x1F05CC82: TCling::AutoParse(char const*) (TCling.cxx:6495)
==2792291==    by 0x1F044564: TClingLookupHelper__AutoParse(char const*) (TCling.cxx:901)
==2792291==    by 0x1EE9F324: ROOT::TMetaUtils::TClingLookupHelper::GetPartiallyDesugaredNameWithScopeHandling(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) (TClingUtils.cxx:581)
==2792291==    by 0x619B73D: TClassEdit::TSplitType::ShortType(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, int) (TClassEdit.cxx:454)
==2792291==    by 0x619CDD6: TClassEdit::GetNormalizedName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::basic_string_view<char, std::char_traits<char> >) (TClassEdit.cxx:871)
==2792291==    by 0x61BF522: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (TClass.cxx:3056)
==2792291==    by 0x61BF1AC: TClass::GetClass(char const*, bool, bool) (TClass.cxx:2971)
==2792291==    by 0x4207098: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==2792291==    by 0x4205C84: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==2792291==    by 0x4F2BD35: edm::BranchDescription::initFromDictionary() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==2792291==    by 0x4F2C1A0: edm::BranchDescription::BranchDescription(edm::BranchType const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::Hash<1> const&, edm::TypeWithDict const&, bool, bool, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==2792291==    by 0x4C7B678: edm::ProductRegistryHelper::addToRegistry(__gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, __gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, edm::ModuleDescription const&, edm::ProductRegistry&, edm::ProductRegistryHelper*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==2792291==    by 0x4C77FEC: edm::ProducerBase::registerProducts(edm::ProducerBase*, edm::ProductRegistry*, edm::ModuleDescription const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==2792291==    by 0x41463ABF: edm::maker::ModuleHolderT<edm::stream::EDProducerAdaptorBase>::registerProductsAndCallbacks(edm::ProductRegistry*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/pluginRecoBTagCombinedPlugins.so)

==2792291== Invalid free() / delete / delete[] / realloc()
==2792291==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x6E724871: ???
==2792291==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2792291==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2792291==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2792291==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2792291==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2792291==  Address 0x639e9cd0 is 0 bytes inside a block of size 26 free'd
==2792291==    at 0x403D4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x6E724871: ???
==2792291==    by 0x1F2F2956: cling::IncrementalExecutor::runAtExitFuncs() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F0490E4: TCling::~TCling() (TCling.cxx:1657)
==2792291==    by 0x1F049325: TCling::~TCling() (TCling.cxx:1665)
==2792291==    by 0x6093CA8: TROOT::~TROOT() (TROOT.cxx:991)
==2792291==    by 0x6090F72: at_exit_of_TROOT() (TROOT.cxx:290)
==2792291==    by 0x7A6E26B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A6E39F: exit (in /usr/lib64/libc-2.28.so)
==2792291==    by 0x7A57D8B: (below main) (in /usr/lib64/libc-2.28.so)
==2792291==  Block was alloc'd at
==2792291==    at 0x403AF01: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2792291==    by 0x512ECA2: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (basic_string.tcc:219)
==2792291==    by 0x512DD9B: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) (basic_string.h:539)
==2792291==    by 0x6E724106: ???
==2792291==    by 0x6E7248AF: ???
==2792291==    by 0x6E72494C: ???
==2792291==    by 0x6E72493F: ???
==2792291==    by 0x2085C661: (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F2F3728: cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F27ED27: cling::Interpreter::executeTransaction(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F302489: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F305248: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F27AF9A: cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/lcg/root/6.27.01-39305256f3f360b1835a05ae811d1e19/lib/libCling.so)
==2792291==    by 0x1F05BD7A: ExecAutoParse(char const*, bool, cling::Interpreter*) (TCling.cxx:6305)
==2792291==    by 0x1F05C602: TCling::AutoParseImplRecurse(char const*, bool) (TCling.cxx:6410)
==2792291==    by 0x1F05C951: TCling::AutoParseImplRecurse(char const*, bool) (TCling.cxx:6446)
==2792291==    by 0x1F05CC82: TCling::AutoParse(char const*) (TCling.cxx:6495)
==2792291==    by 0x1F044564: TClingLookupHelper__AutoParse(char const*) (TCling.cxx:901)
==2792291==    by 0x1EE9F324: ROOT::TMetaUtils::TClingLookupHelper::GetPartiallyDesugaredNameWithScopeHandling(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) (TClingUtils.cxx:581)
==2792291==    by 0x619B73D: TClassEdit::TSplitType::ShortType(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, int) (TClassEdit.cxx:454)
==2792291==    by 0x619CDD6: TClassEdit::GetNormalizedName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::basic_string_view<char, std::char_traits<char> >) (TClassEdit.cxx:871)
==2792291==    by 0x61BF522: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (TClass.cxx:3056)
==2792291==    by 0x61BF1AC: TClass::GetClass(char const*, bool, bool) (TClass.cxx:2971)
==2792291==    by 0x4207098: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==2792291==    by 0x4205C84: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==2792291==    by 0x4F2BD35: edm::BranchDescription::initFromDictionary() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==2792291==    by 0x4F2C1A0: edm::BranchDescription::BranchDescription(edm::BranchType const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::Hash<1> const&, edm::TypeWithDict const&, bool, bool, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==2792291==    by 0x4C7B678: edm::ProductRegistryHelper::addToRegistry(__gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, __gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, edm::ModuleDescription const&, edm::ProductRegistry&, edm::ProductRegistryHelper*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==2792291==    by 0x4C77FEC: edm::ProducerBase::registerProducts(edm::ProducerBase*, edm::ProductRegistry*, edm::ModuleDescription const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==2792291==    by 0x41463ABF: edm::maker::ModuleHolderT<edm::stream::EDProducerAdaptorBase>::registerProductsAndCallbacks(edm::ProductRegistry*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02763/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ROOT6_X_2022-12-15-2300/lib/el8_amd64_gcc11/pluginRecoBTagCombinedPlugins.so)

@Dr15Jones
Copy link
Contributor

So while creating a module we register data products it is creating and one step in that is to call TClass::GetClass. That appears to be triggering the JIT which ultimately registers the same object to be delete at the end of job multiple times.

@makortel
Copy link
Contributor Author

So we should figure out which product type(s) are causing the JIT?

@Dr15Jones
Copy link
Contributor

Dr15Jones commented Dec 20, 2022 •
edited
Loading

So the library that valgrind reports as having the cmsRun module is pluginRecoBTagCombinedPlugins.so which has the following modules

BTagProbabilityToDiscriminator
DeepCMVATagInfoProducer
DeepFlavourJetTagsProducer
DeepNNTagInfoProducer
TrackDeepNNTagInfoProducer

I'm running step3 of workflow 4.44. In configuration for that workflow only the module type not used is TrackDeepNNTagInfoProducer.

After pruning, no additional module types were cut. The following modules (label, type) are used in the job

pfDeepCMVAJetTags DeepFlavourJetTagsProducer
pfDeepCMVATagInfos DeepCMVATagInfoProducer
pfDeepCSVDiscriminatorsJetTags BTagProbabilityToDiscriminator
pfDeepCSVJetTags DeepFlavourJetTagsProducer
pfDeepCSVTagInfos DeepNNTagInfoProducer
pfDeepCSVTagInfosAK8PFPuppiSoftDropSubjets DeepNNTagInfoProducer
pfDeepCSVJetTagsAK8PFPuppiSoftDropSubjets DeepFlavourJetTagsProducer
pfDeepCSVTagInfosAK8Puppi DeepNNTagInfoProducer
pfDeepCSVJetTagsAK8Puppi DeepFlavourJetTagsProducer
pfDeepCSVTagInfosPuppi DeepNNTagInfoProducer
pfDeepCSVJetTagsPuppi DeepFlavourJetTagsProducer
pfDeepCSVDiscriminatorsJetTagsPuppi BTagProbabilityToDiscriminator
pfDeepCSVTagInfosSlimmedDeepFlavour DeepNNTagInfoProducer
pfDeepCSVTagInfosSlimmedPuppiWithDeepTags DeepNNTagInfoProducer
pfParticleNetAK4DiscriminatorsJetTagsSlimmedPuppiWithDeepTags BTagProbabilityToDiscriminator
pfDeepBoostedDiscriminatorsJetTagsSlimmedAK8DeepTags BTagProbabilityToDiscriminator
pfMassDecorrelatedDeepBoostedDiscriminatorsJetTagsSlimmedAK8DeepTags BTagProbabilityToDiscriminator
pfMassDecorrelatedParticleNetDiscriminatorsJetTagsSlimmedAK8DeepTags BTagProbabilityToDiscriminator
pfParticleNetDiscriminatorsJetTagsSlimmedAK8DeepTags BTagProbabilityToDiscriminator

@makortel
Copy link
Contributor Author

Going through

BTagProbabilityToDiscriminator
DeepCMVATagInfoProducer
DeepFlavourJetTagsProducer
DeepNNTagInfoProducer
TrackDeepNNTagInfoProducer

they produce either

cmssw/RecoBTag/Combined/plugins/BTagProbabilityToDiscriminator.cc

Line 85 in 7c0300b

produces<JetTagCollection>(current.name);

or

cmssw/RecoBTag/Combined/plugins/DeepCMVATagInfoProducer.cc

Line 110 in 7c0300b

produces<std::vector<reco::ShallowTagInfo> >();

JetTagCollection is a type alias to

cmssw/DataFormats/BTauReco/interface/JetTag.h

Line 17 in 7c0300b

typedef JetFloatAssociation::Container JetTagCollection;

whose dictionary is declared in

cmssw/DataFormats/JetReco/src/classes_def_3.xml

Line 59 in 7c0300b

<class name="edm::Wrapper<reco::JetFloatAssociation::Container>" />

The dictionary for std::vector<reco::ShallowTagInfo> is defined in

cmssw/DataFormats/BTauReco/src/classes_def.xml

Lines 76 to 79 in 7c0300b

<class name="reco::ShallowTagInfo" ClassVersion="3">
<version ClassVersion="3" checksum="3904302444"/>
</class>
<class name="reco::ShallowTagInfoCollection"/>

Hmm, are we missing a dictionary for reco::JetFloatAssociation::Container itself (I saw only the edm::Wrapper<...> for it)?

@Dr15Jones
Copy link
Contributor

Turning on verbose info from ROOT, it looks like we already defined the appropriate dictionary

Loading TProtoClass for edm::AssociationVector<edm::RefToBaseProd<reco::Jet>,vector<float>,edm::RefToBase<reco::Jet>,unsigned int,edm::helper::AssociationIdenticalKeyReference> - edm::AssociationVector<edm::RefToBaseProd<reco::Jet>,vector<float>,edm::RefToBase<reco::Jet>,unsigned int,edm::helper::AssociationIdenticalKeyReference>

@panoskatsoulis
Copy link
Contributor

Hello all,
recently AlCa is moving the production machines to el8 at P5 and we're testing the new environment.
During testing a similar crash occurred with what you're describing above whiie we're testing L1 O2O, however the O2O jobs have finished succesfully and all the Tags in the DB are updated with correct IOVs.

Thu Mar 30 19:32:03 CEST 2023 : checking O2O
Begin processing the 1st record. Run 399700, Event 1, LumiSection 1 on stream 0 at 30-Mar-2023 19:32:06.688 CEST

-Fatal system signal has occurred during exit
Current TSC key = l1_trg_cosmics2023/v12:l1_trg_rs_cosmics2023/v14
L1TCaloParamsO2ORcd@CaloParams l1_trg_cosmics2023/v12:l1_trg_rs_cosmics2023/v14
L1-O2O-INFO: IOV OK

Above it's a part of the log and the entire log is linked in [1] (conddb site).

More specifically the only module that seems to be resulting in this error is the one in [2], while @francescobrivio also tested the rest O2O sequences and he didn't see similar crashing messages.

As far as I read (briefly) here, this seems to be an issue with freeing up memory at the end and it doesn't affected the job itself. Are there any news on this topic? as it seems that it's a bit stale and we want to understand what is the situation cause AlCa's thinking to move the machines to production some time soon.
FYI: @cms-sw/db-l2 @epalencia @kbunkow

[1] https://cms-conddb.cern.ch/cmsDbBrowser/logs/show_O2O_log/Prep/L1TSubs/2023-03-30%2017:31:57.834130
[2] CondTools/L1TriggerExt/test/l1o2otestanalyzer_cfg.py (in 13_0_0)

@makortel
Copy link
Contributor Author

In general Fatal system signal has occurred during exit can be any kind of crash that occurs after the programs main() function has returned. In this particular case the problem was traced to ROOT dictionaries. Some PRs were made to improve the dictionary declarations (unfortunately not linked here), and we have not seen these problems in ROOT6 or ROOT628 IBs recently (so we probably should close the issue).

Without further information it is very hard to say if your problem has similar origin or not (e.g. we have had a rare Fatal system signal has occurred during exit problem earlier, and its cause was different). I can only suggest to run the job repeatedly through valgrind and/or gdb.

@panoskatsoulis
Copy link
Contributor

Hi @makortel, thx for the suggestion.
Indeed running the job with a debugger will tell us more.
We'll give it a try.

@makortel
Copy link
Contributor Author

More occurrences in CMSSW_13_2_ROOT6_X_2023-06-28-2300 on el8_amd64_gcc11

  • workflow 136.865 step 3
  • workflow 140.114 step 3

@iarspider iarspider mentioned this issue Aug 8, 2023
Merged
@makortel makortel changed the title [ROOT6] Fatal system signal has occurred during exit [ROOT6, ROOT628] Fatal system signal has occurred during exit Aug 9, 2023
@makortel
Copy link
Contributor Author

makortel commented Aug 9, 2023

Occurred in CMSSW_13_3_ROOT628_X_2023-08-08-2300 on el8_amd64_gcc11

  • workflow 140.031 step 3
  • workflow 140.064 step 3

@aandvalenzuela
Copy link
Contributor

Occurred again today in CMSSW_13_3_ROOT628_X_2023-08-09-2300 on el8_amd64_gcc11:

  • workflow 140.008 step 3.
  • workflow 140.044 step 3.
  • workflow 1004.0 step 2.

@aandvalenzuela
Copy link
Contributor

Also in CMSSW_13_3_ROOT6_X_2023-08-09-2300 on el8_amd64_gcc11:

  • workflow 4.52 step 3.
  • workflow 136.87 step 3.

@makortel
Copy link
Contributor Author

Feels like the TCMalloc made this failure mode more frequent.

@makortel makortel reopened this Aug 10, 2023
@makortel
Copy link
Contributor Author

makortel commented Aug 14, 2023 •
edited
Loading

Occurred in CMSSW_13_3_ROOT6_X_2023-08-13-2300

  • workflow 4.22 step 2
  • workflow 139.004 step 3
  • workflow 140.01 step 3

in CMSSW_13_3_ROOT628_X_2023-08-13-2300

  • workflow 136.899 step 2
  • workflow 1004.0 step 2

in CMSSW_13_3_ROOT6_X_2023-08-11-2300

  • workflow 4.34 step 3
  • workflow 4.72 step 3

in CMSSW_13_3_ROOT6_X_2023-08-10-2300

  • workflow 3.0 step 3
  • workflow 4.58 step 2

in CMSSW_13_3_ROOT628_X_2023-08-10-2300

  • workflow 40.075 step 3

@dan131riley
Copy link

Feels like the TCMalloc made this failure mode more frequent.

See also #42468, the stack traces do look different with tcmalloc than we were seeing with jemalloc, so it's not obvious that it's the same problem.

@dan131riley dan131riley mentioned this issue Aug 14, 2023
Closed
@makortel
Copy link
Contributor Author

Occurred in CMSSW_13_3_ROOT6_X_2023-08-14-2300

  • workflow 140.022 step 3

Occurred in CMSSW_13_3_ROOT628_X_2023-08-14-2300

  • workflow 2.0 step 3
  • workflow 4.52 step 3
  • workflow 140.001 step 3

@makortel
Copy link
Contributor Author

Occurred in CMSSW_13_3_ROOT628_X_2023-08-16-2300

  • workflow 136.895 step 3
  • workflow 250206.118 step 3

@makortel
Copy link
Contributor Author

Occurred in CMSSW_13_3_ROOT6_X_2023-08-17-2300

  • workflow 140.028 step 3
  • workflow 140.075 step 3

Occurred in CMSSW_13_3_ROOT628_X_2023-08-17-2300

  • workflow 4.53 step 3
  • workflow 140.068 step 3
  • workflow 140.075 step 3

@pcanal
Copy link
Contributor

pcanal commented Aug 18, 2023

Does root-project/root#13463 improve the behavior?

@makortel
Copy link
Contributor Author

Occurred in CMSSW_13_3_ROOT6_X_2023-08-20-2300

  • workflow 1002.0 step 2
  • workflow 1005.0 step 3

Occurred in CMSSW_13_3_ROOT628_X_2023-08-20-2300

  • workflow 4.37 step 3
  • workflow 140.05 step 3
  • workflow 140.065 step 3
  • workflow 140.104 step 3
  • workflow 1001.0 step 3

@pcanal
Copy link
Contributor

pcanal commented Aug 21, 2023

On a valgrind output on the same test, I see weirdness:

==11011== Mismatched free() / delete / delete []
==11011==    at 0x403CF6C: free (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02798/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==11011==    by 0x5909520: TString::UnLink() const (TString.h:265)
....
==11011==  Address 0x87266e0 is 0 bytes inside a block of size 112 alloc'd
==11011==    at 0x403C0F3: operator new[](unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02798/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==11011==    by 0x6001FCC: TString::Init(int, int) (TString.cxx:283)

and we have:
TString.cxx:283

data = new char[cap+1];

vs
TString.h:265

   void           UnLink() const { if (IsLong()) delete [] fRep.fLong.fData; }

I.e. valgrind got it wrong OR TCMalloc has a deficiency (and/or quirk) that introduces this mismatch.

@makortel
Copy link
Contributor Author

makortel commented Aug 21, 2023 •
edited
Loading

I came across gperftools/gperftools#792 that suggests the Mismatched free() / delete / delete [] report to be false positive, with a comment

It is not good idea to use both tcmalloc and valgrind at same time.

@dan131riley
Copy link

I looked at a bunch of the "mismatched free" warnings and concluded that they are false positives. When I filtered those out it left a handful of invalid frees that do appear to be genuine double-frees in TCling::ResetGlobals(), see #42468 (comment)

@makortel
Copy link
Contributor Author

Occurred in CMSSW_13_3_ROOT6_X_2023-08-22-2300

  • workflow 4.34 step 3
  • workflow 140.008 step 3
  • workflow 140.026 step 3

Occurred in CMSSW_13_3_ROOT628_X_2023-08-22-2300

  • workflow 7.1 step 3
  • workflow 140.051 step 3

@makortel
Copy link
Contributor Author

Occurred in CMSSW_13_3_ROOT6_X_2023-08-24-2300

  • workflow 140.026 step 3
  • workflow 140.047 step 3
  • workflow 140.069 step 3

@makortel
Copy link
Contributor Author

Occurred in CMSSW_13_3_ROOT628_X_2023-08-25-2300

  • workflow 140.002 step 3, although it died at the process exit time with src/tcmalloc.cc:1451] Object was not in-use

Occurred in CMSSW_13_3_ROOT6_X_2023-08-27-2300

  • workflow 140.044 step 3

Occured in CMSSW_13_3_ROOT628_X_2023-08-27-2300

  • workflow 4.76 step 3
  • workflow 140.022, although it died at the process exit time with src/tcmalloc.cc:1451] Object was not in-use

@makortel
Copy link
Contributor Author

Occurred in CMSSW_13_3_ROOT6_X_2023-08-28-2300

  • workflow 140.071 step 3 with src/tcmalloc.cc:1451] Object was not in-use
  • workflow 140.074 step 3 with src/tcmalloc.cc:1451] Object was not in-use

Occurred in CMSSW_13_3_ROOT628_X_2023-08-28-2300

  • workflow 7.1 step 3
  • workflow 140.046 step 3 with src/tcmalloc.cc:1451] Object was not in-use
  • workflow 140.077 step 3 with src/tcmalloc.cc:1451] Object was not in-use

@makortel makortel reopened this Aug 29, 2023
@makortel
Copy link
Contributor Author

makortel commented Aug 29, 2023 •
edited
Loading

The TCMalloc update cms-sw/cmsdist#8635 was merged around 08-24 (I wasn't able to identify the exact IB, for some reason the cmsdist PR does not show up in the per-IB "CMS Dist" PR list). This seems to correspond the appearance of the src/tcmalloc.cc:1451] Object was not in-use messages.

The default was changed back to jemalloc in CMSSW_13_3_X_2023-08-29-1100

@smuzaffar
Copy link
Contributor

@makortel , cms-sw/cmsdist#8635 was first used in 08-24-1100 IB (you can see it by using the "CMS Dist"->"architecture" link of the IB.

@makortel
Copy link
Contributor Author

(you can see it by using the "CMS Dist"->"architecture" link of the IB.

I don't see the link for 08-24-1100 IB
image

@smuzaffar
Copy link
Contributor

click on the Commits to expand it :-)

@smuzaffar
Copy link
Contributor

I will check if we can easily add the externals/cmsdist prs which we were merged for "Default" branch for production arch on the page

@makortel
Copy link
Contributor Author

makortel commented Aug 29, 2023 •
edited
Loading

click on the Commits to expand it :-)

Whoa, didn't know that. Thanks!

@makortel
Copy link
Contributor Author

CMSSW_13_3_X_2023-08-29-2300 IB did not show any failures. Wrt. the previous ROOT IBs, in addition of the changing the default allocator back to jemalloc, the ROOT was updated with cms-sw/cmsdist#8668 (master) and cms-sw/cmsdist#8669 (6.28)

@makortel
Copy link
Contributor Author

Also CMSSW_13_3_X_2023-08-30-2300 didn't show any failures.

@makortel
Copy link
Contributor Author

makortel commented Sep 5, 2023

No new failures in 08-31-2300, 09-01-2300, 09-03-2300.

Given the coincidence with changing back to jemalloc, and observing much lower failure rate with jemalloc than with TCMalloc, we discussed about the option to add a new IB for ROOT master with TCMalloc to build more confidence that this problem has been fixed (or worked around).

@mseidel42 mseidel42 mentioned this issue Apr 4, 2024
Merged
@mseidel42 mseidel42 mentioned this issue Apr 12, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
core-pending pending-signatures
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@makortel @dan131riley @Dr15Jones @smuzaffar @cmsbuild @pcanal @panoskatsoulis @aandvalenzuela