problem with PoolDBOutputService and PPSAssociationCuts

[jan-kaspar]

In CMSSW_12_1_0_pre4, running the command below on lxplus7

cmsRun CondTools/CTPPS/test/write_PPSAssociationCuts_cfg.py

leads to a crash:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thu Oct 14 10:36:14 CEST 2021
Thread 2 (Thread 0x7f9697a03700 (LWP 17061) "cmsRun"):
#0  0x00007f96b48071d9 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f96ae7e35a7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_0_pre4/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#2  0x00007f96ae7e423a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_0_pre4/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  0x00007f96b4e00af0 in std::execute_native_thread_routine (__p=0x7f9697bc10d0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007f96b47ffea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f96b45279fd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f96b2683540 (LWP 17041) "cmsRun"):
#0  0x00007f96b451cccd in poll () from /lib64/libc.so.6
#1  0x00007f96ae7e39d7 in full_read.constprop () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_0_pre4/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#2  0x00007f96ae7e430c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_0_pre4/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  0x00007f96ae7e77bb in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_0_pre4/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f96968ce751 in std::_Rb_tree<unsigned int, std::pair<unsigned int const, PPSAssociationCuts::CutsPerArm>, std::_Select1st<std::pair<unsigned int const, PPSAssociationCuts::CutsPerArm> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, PPSAssociationCuts::CutsPerArm> > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, PPSAssociationCuts::CutsPerArm> >*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_0_pre4/lib/slc7_amd64_gcc900/libCondFormatsPPSObjects.so
#6  0x00007f9696c8b206 in std::_Sp_counted_ptr_inplace<PPSAssociationCutsESSource, std::allocator<PPSAssociationCutsESSource>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_0_pre4/lib/slc7_amd64_gcc900/pluginCalibPPSESProducersPlugins.so
#7  0x00007f96b6ecc7ad in std::_Sp_counted_deleter<edm::eventsetup::EventSetupRecordProvider*, std::default_delete<edm::eventsetup::EventSetupRecordProvider>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_0_pre4/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#8  0x00007f96b6ec9055 in edm::eventsetup::EventSetupProvider::~EventSetupProvider() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_0_pre4/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#9  0x00007f96b6ee1f32 in std::_Sp_counted_deleter<edm::eventsetup::EventSetupProvider*, std::default_delete<edm::eventsetup::EventSetupProvider>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_0_pre4/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#10 0x000000000040f4f8 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#11 0x00007f96b6e838c6 in edm::EventProcessor::~EventProcessor() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_0_pre4/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#12 0x000000000040b851 in (anonymous namespace)::EventProcessorWithSentry::~EventProcessorWithSentry() ()
#13 0x000000000040b649 in main ()

Current Modules:

Module: none (crashed)

A fatal system signal has occurred: segmentation violation

By commenting this line
https://github.com/cms-sw/cmssw/blob/master/CondTools/CTPPS/plugins/WritePPSAssociationCuts.cc#L44
the crash is avoided. Thus I guess this comes from some kind of interaction between PoolDBOutputService and PPSAssociationCuts.

By inserting print outs, I could check that WritePPSAssociationCuts::analyze gets successfully to the end. Thus I guess the problem occurs at destruction time.

Any help appreciated! Many thanks in advance!

[cmsbuild]

A new Issue was created by @jan-kaspar .

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[mmusich]

diff --git a/CondTools/CTPPS/plugins/WritePPSAssociationCuts.cc b/CondTools/CTPPS/plugins/WritePPSAssociationCuts.cc
index 97f0eb3d4bb..6d2dbe84b24 100644
--- a/CondTools/CTPPS/plugins/WritePPSAssociationCuts.cc
+++ b/CondTools/CTPPS/plugins/WritePPSAssociationCuts.cc
@@ -41,7 +41,7 @@ void WritePPSAssociationCuts::analyze(const edm::Event &iEvent, const edm::Event
   // store the data in a DB object
   edm::Service<cond::service::PoolDBOutputService> poolDbService;
   if (poolDbService.isAvailable()) {
-    poolDbService->writeOne(&ppsAssociationCuts, poolDbService->currentTime(), "PPSAssociationCutsRcd");
+    poolDbService->writeOneIOV(&ppsAssociationCuts, poolDbService->currentTime(), "PPSAssociationCutsRcd");
   } else {
     throw cms::Exception("WritePPSAssociationCuts") << "PoolDBService required.";
   }

will do.
See #35556.

[makortel]

assign db

[cmsbuild]

New categories assigned: db

@ggovi,@francescobrivio,@malbouis,@tvami you have been requested to review this Pull request/Issue and eventually sign? Thanks

[jan-kaspar]

Many thanks @mmusich !

@wpcarvalho or @grzanka would you have possibly a chance to open a PR with the proposed fix, please?

[tvami]

I submitted this PR: #35669
I'm sorry that this is happening, Yuan was supposed to inform everybody a week ago that people have to migrate to the new method. He even compiled the list here cms-AlCaDB/AlCaTools#28 he now says he'll write the email today...

[jan-kaspar]

Hi @tvami no problem and many thanks for the PR!