Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

segfaults in CAHitQuadrupletEDProducer #36808

Closed
dan131riley opened this issue Jan 26, 2022 · 12 comments
Closed

segfaults in CAHitQuadrupletEDProducer #36808

dan131riley opened this issue Jan 26, 2022 · 12 comments
Labels
pending-signatures reconstruction-pending

Comments

@dan131riley
Copy link

Two crashes in CAHitQuadrupletEDProducer, wf 1030.0 and 140.56:

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/raw/slc7_amd64_gcc10/CMSSW_12_3_X_2022-01-25-2300/pyRelValMatrixLogs/run/1030.0_RunHLTPhy2017B+RunHLTPhy2017B+TIER0EXPHPBS+ALCASPLITHPBS+ALCAHARVDHPBS+ALCAHARVDHPBSLOWPU/step2_RunHLTPhy2017B+RunHLTPhy2017B+TIER0EXPHPBS+ALCASPLITHPBS+ALCAHARVDHPBS+ALCAHARVDHPBSLOWPU.log

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/raw/cs8_amd64_gcc11/CMSSW_12_3_X_2022-01-25-2300/pyRelValMatrixLogs/run/140.56_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18/step2_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18.log

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Wed Jan 26 02:08:18 CET 2022
hread 10 (Thread 0x2b9ea1800700 (LWP 10629) "cmsRun"):
#0  0x00002b9e53eda6a8 in nanosleep () from /lib64/libc.so.6
#1  0x00002b9e53eda5ae in sleep () from /lib64/libc.so.6
#2  0x00002b9e589e0390 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b9f0e3a6e11 in TrajectorySegmentBuilder::updateTrajectory(TempTrajectory&, TrajectoryMeasurement) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#5  0x00002b9f0e3a7849 in TrajectorySegmentBuilder::updateCandidates(TempTrajectory const&, std::vector<TrajectoryMeasurement, std::allocator<TrajectoryMeasurement> > const&, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#6  0x00002b9f0e3a900b in TrajectorySegmentBuilder::addGroup(TempTrajectory const&, __gnu_cxx::__normal_iterator<TrajectoryMeasurementGroup const*, std::vector<TrajectoryMeasurementGroup, std::allocator<TrajectoryMeasurementGroup> > >, __gnu_cxx::__normal_iterator<TrajectoryMeasurementGroup const*, std::vector<TrajectoryMeasurementGroup, std::allocator<TrajectoryMeasurementGroup> > >) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#7  0x00002b9f0e3aa3e5 in TrajectorySegmentBuilder::segments(TrajectoryStateOnSurface) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#8  0x00002b9f0e3936e2 in GroupedCkfTrajectoryBuilder::advanceOneLayer(TrajectorySeed const&, TempTrajectory&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#9  0x00002b9f0e3954fa in GroupedCkfTrajectoryBuilder::groupedLimitedCandidates(TrajectorySeed const&, TempTrajectory const&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#10 0x00002b9f0e397536 in GroupedCkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector<Trajectory, std::allocator<Trajectory> >&, unsigned int&, TrajectoryFilter const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#11 0x00002b9f0dd2c99a in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libRecoTrackerCkfPattern.so
#12 0x00002b9e517c8fad in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#13 0x00002b9e517b263f in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#14 0x00002b9e5170e665 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#15 0x00002b9e5170ea13 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#16 0x00002b9e5171105a in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#17 0x00002b9e514a27a5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreConcurrency.so
#18 0x00002b9e52f1b5bc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x2b9f54feaf00, waiter=..., this=0x2b9e559ed500) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
#19 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x2b9e559ed500) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
#20 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/arena.cpp:138
#21 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/market.cpp:597
#22 0x00002b9e52f1d6d4 in tbb::detail::r1::rml::private_worker::run (this=0x2b9e577ed000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/private_server.cpp:267
#23 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b9e577ed000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/private_server.cpp:221
#24 0x00002b9e53bc817a in start_thread () from /lib64/libpthread.so.0
#25 0x00002b9e53e19d83 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x2b9ea0801700 (LWP 10628) "cmsRun"):
#0  0x00002b9e53eda6a8 in nanosleep () from /lib64/libc.so.6
#1  0x00002b9e53eda5ae in sleep () from /lib64/libc.so.6
#2  0x00002b9e589e0390 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b9e83c622c8 in ProxyStripTopology::positionCorrection(Topology::LocalTrackPred const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libGeometryTrackerGeometryBuilder.so
#5  0x00002b9e83c624e5 in ProxyStripTopology::localPosition(float, Topology::LocalTrackPred const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libGeometryTrackerGeometryBuilder.so
#6  0x00002b9e83928881 in StripCPEfromTrackAngle::localParameters(SiStripCluster const&, StripCPE::AlgoParam const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libRecoLocalTrackerSiStripRecHitConverter.so
#7  0x00002b9e83879d09 in bool TkStripMeasurementDet::filteredRecHits<edm::Ref<edmNew::DetSetVector<SiStripCluster>, SiStripCluster, edmNew::DetSetVector<SiStripCluster>::FindForDetSetVector> >(edm::Ref<edmNew::DetSetVector<SiStripCluster>, SiStripCluster, edmNew::DetSetVector<SiStripCluster>::FindForDetSetVector> const&, StripCPE::AlgoParam const&, TrajectoryStateOnSurface const&, MeasurementEstimator const&, std::vector<bool, std::allocator<bool> > const&, std::vector<SiStripRecHit2D, std::allocator<SiStripRecHit2D> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so
#8  0x00002b9e83877925 in TkStripMeasurementDet::simpleRecHits(TrajectoryStateOnSurface const&, MeasurementEstimator const&, MeasurementTrackerEvent const&, std::vector<SiStripRecHit2D, std::allocator<SiStripRecHit2D> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so
#9  0x00002b9e8386b834 in void TkGluedMeasurementDet::doubleMatch<TkGluedMeasurementDet::HitCollectorForFastMeasurements>(TrajectoryStateOnSurface const&, MeasurementTrackerEvent const&, TkGluedMeasurementDet::HitCollectorForFastMeasurements&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so
#10 0x00002b9e83867aa7 in TkGluedMeasurementDet::measurements(TrajectoryStateOnSurface const&, MeasurementEstimator const&, MeasurementTrackerEvent const&, tracking::TempMeasurements&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so
#11 0x00002b9e838f5f17 in LayerMeasurements::groupedMeasurements(DetLayer const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libTrackingToolsMeasurementDet.so
#12 0x00002b9f0e3aa3c1 in TrajectorySegmentBuilder::segments(TrajectoryStateOnSurface) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#13 0x00002b9f0e3936e2 in GroupedCkfTrajectoryBuilder::advanceOneLayer(TrajectorySeed const&, TempTrajectory&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#14 0x00002b9f0e3954fa in GroupedCkfTrajectoryBuilder::groupedLimitedCandidates(TrajectorySeed const&, TempTrajectory const&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#15 0x00002b9f0e397536 in GroupedCkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector<Trajectory, std::allocator<Trajectory> >&, unsigned int&, TrajectoryFilter const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#16 0x00002b9f0dd2c99a in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libRecoTrackerCkfPattern.so
#17 0x00002b9e517c8fad in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#18 0x00002b9e517b263f in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#19 0x00002b9e5170e665 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#20 0x00002b9e5170ea13 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#21 0x00002b9e5171105a in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#22 0x00002b9e514a27a5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreConcurrency.so
#23 0x00002b9e52f1b5bc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x2b9ea183a400, waiter=..., this=0x2b9e559ed400) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
#24 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x2b9e559ed400) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
#25 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/arena.cpp:138
#26 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/market.cpp:597
#27 0x00002b9e52f1d6d4 in tbb::detail::r1::rml::private_worker::run (this=0x2b9e577ed100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/private_server.cpp:267
#28 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b9e577ed100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/private_server.cpp:221
#29 0x00002b9e53bc817a in start_thread () from /lib64/libpthread.so.0
#30 0x00002b9e53e19d83 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x2b9e9fe00700 (LWP 10627) "cmsRun"):
#0  0x00002b9e53eda6a8 in nanosleep () from /lib64/libc.so.6
#1  0x00002b9e53eda5ae in sleep () from /lib64/libc.so.6
#2  0x00002b9e589e0390 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b9e5bec2750 in TkRadialStripTopology::yDistanceToIntersection(float) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libGeometryCommonTopologies.so
#5  0x00002b9e5bec281c in TkRadialStripTopology::localPosition(Point2DBase<float, MeasurementTag> const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libGeometryCommonTopologies.so
#6  0x00002b9e83c62b08 in ProxyStripTopology::localPosition(Point2DBase<float, MeasurementTag> const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libGeometryTrackerGeometryBuilder.so
#7  0x00002b9e8386c3a8 in void TkGluedMeasurementDet::doubleMatch<TkGluedMeasurementDet::HitCollectorForFastMeasurements>(TrajectoryStateOnSurface const&, MeasurementTrackerEvent const&, TkGluedMeasurementDet::HitCollectorForFastMeasurements&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so
#8  0x00002b9e83867aa7 in TkGluedMeasurementDet::measurements(TrajectoryStateOnSurface const&, MeasurementEstimator const&, MeasurementTrackerEvent const&, tracking::TempMeasurements&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so
#9  0x00002b9f0e3a8853 in TrajectorySegmentBuilder::redoMeasurements(TempTrajectory const&, DetGroup const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#10 0x00002b9f0e3a9298 in TrajectorySegmentBuilder::addGroup(TempTrajectory const&, __gnu_cxx::__normal_iterator<TrajectoryMeasurementGroup const*, std::vector<TrajectoryMeasurementGroup, std::allocator<TrajectoryMeasurementGroup> > >, __gnu_cxx::__normal_iterator<TrajectoryMeasurementGroup const*, std::vector<TrajectoryMeasurementGroup, std::allocator<TrajectoryMeasurementGroup> > >) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#11 0x00002b9f0e3a90f7 in TrajectorySegmentBuilder::addGroup(TempTrajectory const&, __gnu_cxx::__normal_iterator<TrajectoryMeasurementGroup const*, std::vector<TrajectoryMeasurementGroup, std::allocator<TrajectoryMeasurementGroup> > >, __gnu_cxx::__normal_iterator<TrajectoryMeasurementGroup const*, std::vector<TrajectoryMeasurementGroup, std::allocator<TrajectoryMeasurementGroup> > >) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#12 0x00002b9f0e3aa3e5 in TrajectorySegmentBuilder::segments(TrajectoryStateOnSurface) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#13 0x00002b9f0e3936e2 in GroupedCkfTrajectoryBuilder::advanceOneLayer(TrajectorySeed const&, TempTrajectory&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#14 0x00002b9f0e3954fa in GroupedCkfTrajectoryBuilder::groupedLimitedCandidates(TrajectorySeed const&, TempTrajectory const&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#15 0x00002b9f0e397536 in GroupedCkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector<Trajectory, std::allocator<Trajectory> >&, unsigned int&, TrajectoryFilter const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#16 0x00002b9f0dd2c99a in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libRecoTrackerCkfPattern.so
#17 0x00002b9e517c8fad in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#18 0x00002b9e517b263f in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#19 0x00002b9e5170e665 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#20 0x00002b9e5170ea13 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#21 0x00002b9e5171105a in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#22 0x00002b9e514a27a5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreConcurrency.so
#23 0x00002b9e52f1b5bc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x2b9f54fb0800, waiter=..., this=0x2b9e559ed480) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
#24 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x2b9e559ed480) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
#25 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/arena.cpp:138
#26 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/market.cpp:597
#27 0x00002b9e52f1d6d4 in tbb::detail::r1::rml::private_worker::run (this=0x2b9e577ed080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/private_server.cpp:267
#28 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b9e577ed080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/private_server.cpp:221
#29 0x00002b9e53bc817a in start_thread () from /lib64/libpthread.so.0
#30 0x00002b9e53e19d83 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b9e54c39240 (LWP 27196) "cmsRun"):
#0  0x00002b9e53f04ab1 in poll () from /lib64/libc.so.6
#1  0x00002b9e589e07ff in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#2  0x00002b9e589e115c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  0x00002b9e589e399b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b9f1697cb58 in (anonymous namespace)::fillNtuplets(RegionsSeedingHitSets::RegionFiller&, OrderedHitSeeds const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoPixelVertexingPixelTripletsPlugins.so
#6  0x00002b9f16985ce7 in CAHitNtupletEDProducerT<CAHitQuadrupletGenerator>::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/pluginRecoPixelVertexingPixelTripletsPlugins.so
#7  0x00002b9e517c8fad in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#8  0x00002b9e517b263f in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#9  0x00002b9e5170e665 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#10 0x00002b9e5170ea13 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#11 0x00002b9e5171105a in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#12 0x00002b9e514a27a5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreConcurrency.so
#13 0x00002b9e52f22a59 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x2b9ea22d6d00, this=0x2b9e559ed380) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x2b9e559ed380) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
#15 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/task_dispatcher.cpp:168
#16 0x00002b9e5167ed2f in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#17 0x00002b9e51689317 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02717/cs8_amd64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/cs8_amd64_gcc11/libFWCoreFramework.so
#18 0x000000000040a18d in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#19 0x00002b9e52f10898 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/cs8_amd64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/tbb-v2021.4.0/src/tbb/arena.cpp:698
#20 0x000000000040afd9 in main::{lambda()#1}::operator()() const ()
#21 0x00000000004096ec in main ()

Current Modules:

Module: CAHitQuadrupletEDProducer:hiConformalPixelTracksPhase1HitQuadrupletsCA (crashed)
Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates
Module: CkfTrackCandidateMaker:initialStepTrackCandidates
Module: CkfTrackCandidateMaker:initialStepTrackCandidates

A fatal system signal has occurred: segmentation violation

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Wed Jan 26 07:46:39 CET 2022
Thread 10 (Thread 0x2b09d5400700 (LWP 26592) "cmsRun"):
#0  0x00002b098808c9fd in nanosleep () from /lib64/libc.so.6
#1  0x00002b098808c894 in sleep () from /lib64/libc.so.6
#2  0x00002b098dc10ca0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b09df647c39 in trajectoryToResiduals(Trajectory const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libRecoTrackerTrackProducer.so
#5  0x00002b09df63b093 in KfTrackProducerBase::putInEvt(edm::Event&, Propagator const*, MeasurementTracker const*, std::unique_ptr<edm::OwnVector<TrackingRecHit, edm::ClonePolicy<TrackingRecHit> >, std::default_delete<edm::OwnVector<TrackingRecHit, edm::ClonePolicy<TrackingRecHit> > > >&, std::unique_ptr<std::vector<reco::Track, std::allocator<reco::Track> >, std::default_delete<std::vector<reco::Track, std::allocator<reco::Track> > > >&, std::unique_ptr<std::vector<reco::TrackExtra, std::allocator<reco::TrackExtra> >, std::default_delete<std::vector<reco::TrackExtra, std::allocator<reco::TrackExtra> > > >&, std::unique_ptr<std::vector<Trajectory, std::allocator<Trajectory> >, std::default_delete<std::vector<Trajectory, std::allocator<Trajectory> > > >&, std::unique_ptr<std::vector<int, std::allocator<int> >, std::default_delete<std::vector<int, std::allocator<int> > > >&, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&, TransientTrackingRecHitBuilder const*, TrackerTopology const*, int) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libRecoTrackerTrackProducer.so
#6  0x00002b09df51cf69 in TrackProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginRecoTrackerTrackProducerPlugins.so
#7  0x00002b0985a5b4c3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#8  0x00002b0985a447df in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#9  0x00002b098599fc15 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#10 0x00002b098599ff0b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#11 0x00002b09859a24f5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#12 0x00002b09857367d5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so
#13 0x00002b09871ac06c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x2b098aca6a00, waiter=..., this=0x2b098aded500) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x2b098aded500) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
#15 tbb::detail::r1::arena::process (this=<optimized out>, tls=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/arena.cpp:138
#16 0x00002b09871b85b3 in tbb::detail::r1::market::process (j=..., this=0x2b098adef580) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/market.cpp:597
#17 tbb::detail::r1::rml::private_worker::run (this=0x2b098cc70000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/private_server.cpp:267
#18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b098cc70000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/private_server.cpp:221
#19 0x00002b0987db2ea5 in start_thread () from /lib64/libpthread.so.0
#20 0x00002b09880c5b0d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x2b09d4401700 (LWP 26591) "cmsRun"):
#0  0x00002b09880baddd in poll () from /lib64/libc.so.6
#1  0x00002b098dc1110f in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#2  0x00002b098dc11a9c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  0x00002b098dc143eb in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b09b6035778 in CAHitQuadrupletGenerator::hitNtuplets(IntermediateHitDoublets const&, std::vector<OrderedHitSeeds, std::allocator<OrderedHitSeeds> >&, SeedingLayerSetsHits const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libRecoPixelVertexingPixelTriplets.so
#6  0x00002b09e33d52fa in CAHitNtupletEDProducerT<CAHitQuadrupletGenerator>::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginRecoPixelVertexingPixelTripletsPlugins.so
#7  0x00002b0985a5b4c3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#8  0x00002b0985a447df in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#9  0x00002b098599fc15 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#10 0x00002b098599ff0b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#11 0x00002b09859a24f5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#12 0x00002b09857367d5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so
#13 0x00002b09871ac06c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x2b09d5f2a100, waiter=..., this=0x2b098aded480) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x2b098aded480) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
#15 tbb::detail::r1::arena::process (this=<optimized out>, tls=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/arena.cpp:138
#16 0x00002b09871b85b3 in tbb::detail::r1::market::process (j=..., this=0x2b098adef580) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/market.cpp:597
#17 tbb::detail::r1::rml::private_worker::run (this=0x2b098cc70100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/private_server.cpp:267
#18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b098cc70100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/private_server.cpp:221
#19 0x00002b0987db2ea5 in start_thread () from /lib64/libpthread.so.0
#20 0x00002b09880c5b0d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x2b09d3a00700 (LWP 26590) "cmsRun"):
#0  0x00002b098808c9fd in nanosleep () from /lib64/libc.so.6
#1  0x00002b098808c894 in sleep () from /lib64/libc.so.6
#2  0x00002b098dc10ca0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b09e2f4cf67 in void Traj2TrackHits::split<__gnu_cxx::__normal_iterator<TrajectoryMeasurement const*, std::vector<TrajectoryMeasurement, std::allocator<TrajectoryMeasurement> > > >(__gnu_cxx::__normal_iterator<TrajectoryMeasurement const*, std::vector<TrajectoryMeasurement, std::allocator<TrajectoryMeasurement> > >, __gnu_cxx::__normal_iterator<TrajectoryMeasurement const*, std::vector<TrajectoryMeasurement, std::allocator<TrajectoryMeasurement> > >, edm::OwnVector<TrackingRecHit, edm::ClonePolicy<TrackingRecHit> >&, bool) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libRecoTrackerCkfPattern.so
#5  0x00002b09e2f4d3f5 in Traj2TrackHits::operator()(Trajectory const&, edm::OwnVector<TrackingRecHit, edm::ClonePolicy<TrackingRecHit> >&, bool) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libRecoTrackerCkfPattern.so
#6  0x00002b09e2f4072d in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libRecoTrackerCkfPattern.so
#7  0x00002b0985a5b4c3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#8  0x00002b0985a447df in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#9  0x00002b098599fc15 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#10 0x00002b098599ff0b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#11 0x00002b09859a24f5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#12 0x00002b09857367d5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so
#13 0x00002b09871ac06c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x2b09d5f16300, waiter=..., this=0x2b098aded400) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x2b098aded400) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
#15 tbb::detail::r1::arena::process (this=<optimized out>, tls=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/arena.cpp:138
#16 0x00002b09871b85b3 in tbb::detail::r1::market::process (j=..., this=0x2b098adef580) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/market.cpp:597
#17 tbb::detail::r1::rml::private_worker::run (this=0x2b098cc70080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/private_server.cpp:267
#18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b098cc70080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/private_server.cpp:221
#19 0x00002b0987db2ea5 in start_thread () from /lib64/libpthread.so.0
#20 0x00002b09880c5b0d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b098a149b00 (LWP 26399) "cmsRun"):
#0  0x00002b098808c9fd in nanosleep () from /lib64/libc.so.6
#1  0x00002b098808c894 in sleep () from /lib64/libc.so.6
#2  0x00002b098dc10ca0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b09b604b642 in CellularAutomaton::findTriplets(std::vector<HitDoublets const*, std::allocator<HitDoublets const*> > const&, std::vector<std::vector<unsigned int, std::allocator<unsigned int> >, std::allocator<std::vector<unsigned int, std::allocator<unsigned int> > > >&, TrackingRegion const&, CACut const&, CACut const&, float) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libRecoPixelVertexingPixelTriplets.so
#5  0x00002b09b6043d84 in CAHitTripletGenerator::hitNtuplets(IntermediateHitDoublets const&, std::vector<OrderedHitSeeds, std::allocator<OrderedHitSeeds> >&, SeedingLayerSetsHits const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libRecoPixelVertexingPixelTriplets.so
#6  0x00002b09e33d5b9a in CAHitNtupletEDProducerT<CAHitTripletGenerator>::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginRecoPixelVertexingPixelTripletsPlugins.so
#7  0x00002b0985a5b4c3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#8  0x00002b0985a447df in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#9  0x00002b098599fc15 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#10 0x00002b098599ff0b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#11 0x00002b09859a24f5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#12 0x00002b09857367d5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so
#13 0x00002b09871beb8c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x2b09d5f64200, this=0x2b098aded380) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x2b098aded380) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
#15 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.cpp:168
#16 0x00002b0985910908 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#17 0x00002b098591b63b in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#18 0x000000000040a266 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#19 0x00002b09871ad15b in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/arena.cpp:698
#20 0x000000000040b094 in main::{lambda()#1}::operator()() const ()
#21 0x000000000040971c in main ()

Current Modules:

Module: CAHitQuadrupletEDProducer:lowPtQuadStepHitQuadruplets (crashed)
Module: CkfTrackCandidateMaker:pixelPairStepTrackCandidates
Module: TrackProducer:tobTecStepTracks
Module: CAHitTripletEDProducer:detachedTripletStepHitTriplets

A fatal system signal has occurred: segmentation violation
@cmsbuild
Copy link
Contributor

A new Issue was created by @dan131riley Dan Riley.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor

assign reconstruction

@cmsbuild
Copy link
Contributor

New categories assigned: reconstruction

@slava77,@jpata,@clacaputo you have been requested to review this Pull request/Issue and eventually sign? Thanks

@makortel
Copy link
Contributor

I'd guess the problem is the same as in #36804.

@makortel makortel mentioned this issue Jan 26, 2022
Closed
@missirol
Copy link
Contributor

The HLT-Validation tests in IB CMSSW_12_3_X_2022-01-25-2300 failed, from what looks like a crash from CAHitQuadrupletEDProducer [1].

I'm guessing that failure is related to the issue discussed here (cc: @Martin-Grunewald).

%MSG-i ThreadStreamSetup:  (NoModuleName) 26-Jan-2022 08:59:08 UTC pre-events
setting # threads 4
setting # streams 4
%MSG
%MSG-w CUDAService:  (NoModuleName) 26-Jan-2022 08:59:09 UTC pre-events
Failed to initialize the CUDA runtime.
Disabling the CUDAService.
%MSG
26-Jan-2022 08:59:14 UTC  Initiating request to open file file:../RelVal_Raw_HIon_MC.root
26-Jan-2022 08:59:16 UTC  Successfully opened file file:../RelVal_Raw_HIon_MC.root
%MSG-e TkDetLayers:  PostModuleEvent 26-Jan-2022 08:59:24 UTC Run: 1 Event: 3
 ForwardDiskSectorBuilderFromDet: Trying to build Petal Wedge from Dets at different z positions !! Delta_z = -0.951241
%MSG


A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Wed Jan 26 08:59:54 UTC 2022
Thread 5 (Thread 0x2b32fde00700 (LWP 75140) "cmsRun"):
#0  0x00002b32b58929fd in nanosleep () from /lib64/libc.so.6
#1  0x00002b32b5892894 in sleep () from /lib64/libc.so.6
#2  0x00002b32bbb1dca0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b33060c63d4 in CaloTowersCreationAlgo::hadShwrPos(CaloTowerDetId, float) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginRecoLocalCaloCaloTowersCreator.so
#5  0x00002b33060c8751 in CaloTowersCreationAlgo::convert(CaloTowerDetId const&, CaloTowersCreationAlgo::MetaTower const&, edm::SortedCollection<CaloTower, edm::StrictWeakOrdering<CaloTower> >&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginRecoLocalCaloCaloTowersCreator.so
#6  0x00002b33060c8b33 in CaloTowersCreationAlgo::finish(edm::SortedCollection<CaloTower, edm::StrictWeakOrdering<CaloTower> >&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginRecoLocalCaloCaloTowersCreator.so
#7  0x00002b33060d2135 in CaloTowersCreator::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginRecoLocalCaloCaloTowersCreator.so
#8  0x00002b32b32614c3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#9  0x00002b32b324a7df in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#10 0x00002b32b31a5c15 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#11 0x00002b32b31a5f0b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#12 0x00002b32b31a84f5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#13 0x00002b32b2f3c7d5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so
#14 0x00002b32b49b206c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x2b32b84fee00, waiter=..., this=0x2b32b85eea00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
#15 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x2b32b85eea00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
#16 tbb::detail::r1::arena::process (this=<optimized out>, tls=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/arena.cpp:138
#17 0x00002b32b49be5b3 in tbb::detail::r1::market::process (j=..., this=0x2b32b85ef580) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/market.cpp:597
#18 tbb::detail::r1::rml::private_worker::run (this=0x2b32ba535000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/private_server.cpp:267
#19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b32ba535000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/private_server.cpp:221
#20 0x00002b32b55b8ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00002b32b58cbb0d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x2b32fceff700 (LWP 75139) "cmsRun"):
#0  0x00002b32b58929fd in nanosleep () from /lib64/libc.so.6
#1  0x00002b32b5892894 in sleep () from /lib64/libc.so.6
#2  0x00002b32bbb1dca0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b32b2e8e531 in edm::DataFrameContainer::sort() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libDataFormatsCommon.so
#5  0x00002b3305ad52f7 in HcalRawToDigi::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginEventFilterHcalRawToDigiPlugins.so
#6  0x00002b32b32614c3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#7  0x00002b32b324a7df in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#8  0x00002b32b31a5c15 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#9  0x00002b32b31a5f0b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#10 0x00002b32b31a84f5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#11 0x00002b32b30eca95 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#12 0x00002b32b49b206c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x2b32b8514b00, waiter=..., this=0x2b32b85ee900) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
#13 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x2b32b85ee900) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
#14 tbb::detail::r1::arena::process (this=<optimized out>, tls=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/arena.cpp:138
#15 0x00002b32b49be5b3 in tbb::detail::r1::market::process (j=..., this=0x2b32b85ef580) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/market.cpp:597
#16 tbb::detail::r1::rml::private_worker::run (this=0x2b32ba535100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/private_server.cpp:267
#17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b32ba535100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/private_server.cpp:221
#18 0x00002b32b55b8ea5 in start_thread () from /lib64/libpthread.so.0
#19 0x00002b32b58cbb0d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x2b32fc4fe700 (LWP 75138) "cmsRun"):
#0  0x00002b32b58c0ddd in poll () from /lib64/libc.so.6
#1  0x00002b32bbb1e10f in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#2  0x00002b32bbb1ea9c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  0x00002b32bbb213eb in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b32e2991778 in CAHitQuadrupletGenerator::hitNtuplets(IntermediateHitDoublets const&, std::vector<OrderedHitSeeds, std::allocator<OrderedHitSeeds> >&, SeedingLayerSetsHits const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libRecoPixelVertexingPixelTriplets.so
#6  0x00002b33065112fa in CAHitNtupletEDProducerT<CAHitQuadrupletGenerator>::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginRecoPixelVertexingPixelTripletsPlugins.so
#7  0x00002b32b32614c3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#8  0x00002b32b324a7df in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#9  0x00002b32b31a5c15 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#10 0x00002b32b31a5f0b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#11 0x00002b32b31a84f5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#12 0x00002b32b2f3c7d5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so
#13 0x00002b32b49b206c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x2b32b8505c00, waiter=..., this=0x2b32b85ee980) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x2b32b85ee980) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
#15 tbb::detail::r1::arena::process (this=<optimized out>, tls=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/arena.cpp:138
#16 0x00002b32b49be5b3 in tbb::detail::r1::market::process (j=..., this=0x2b32b85ef580) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/market.cpp:597
#17 tbb::detail::r1::rml::private_worker::run (this=0x2b32ba535080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/private_server.cpp:267
#18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b32ba535080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/private_server.cpp:221
#19 0x00002b32b55b8ea5 in start_thread () from /lib64/libpthread.so.0
#20 0x00002b32b58cbb0d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x2b32d8eab700 (LWP 75088) "cmsRun"):
#0  0x00002b32b55c01d9 in waitpid () from /lib64/libpthread.so.0
#1  0x00002b32bbb1de57 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#2  0x00002b32bbb1e9ca in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  0x00002b32b5194f90 in std::execute_native_thread_routine (__p=0x2b32cc719810) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00002b32b55b8ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00002b32b58cbb0d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b32b794fb00 (LWP 75022) "cmsRun"):
#0  0x00002b32b58929fd in nanosleep () from /lib64/libc.so.6
#1  0x00002b32b5892894 in sleep () from /lib64/libc.so.6
#2  0x00002b32bbb1dca0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b32e261e0d8 in SiPixelGainCalibrationOffline::getPed(int const&, int const&, std::pair<__gnu_cxx::__normal_iterator<char const*, std::vector<char, std::allocator<char> > >, __gnu_cxx::__normal_iterator<char const*, std::vector<char, std::allocator<char> > > > const&, int const&, bool&, bool&) const () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libCondFormatsSiPixelObjects.so
#5  0x00002b32e4a11c84 in SiPixelGainCalibrationServicePayloadGetter<SiPixelGainCalibrationOffline, SiPixelGainCalibrationOfflineRcd>::getPedestalByPixel(unsigned int const&, int const&, int const&, bool&, bool&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libCalibTrackerSiPixelESProducers.so
#6  0x00002b32e4a10e71 in SiPixelGainCalibrationOfflineService::isDead(unsigned int const&, int const&, int const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libCalibTrackerSiPixelESProducers.so
#7  0x00002b32e4a15775 in SiPixelGainCalibrationServiceBase::calibrate(unsigned int, __gnu_cxx::__normal_iterator<PixelDigi const*, std::vector<PixelDigi, std::allocator<PixelDigi> > >, __gnu_cxx::__normal_iterator<PixelDigi const*, std::vector<PixelDigi, std::allocator<PixelDigi> > >, float, float, int*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libCalibTrackerSiPixelESProducers.so
#8  0x00002b33061e510e in PixelThresholdClusterizer::copy_to_buffer(__gnu_cxx::__normal_iterator<PixelDigi const*, std::vector<PixelDigi, std::allocator<PixelDigi> > >, __gnu_cxx::__normal_iterator<PixelDigi const*, std::vector<PixelDigi, std::allocator<PixelDigi> > >) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so
#9  0x00002b33061e9068 in void PixelThresholdClusterizer::clusterizeDetUnitT<edm::DetSet<PixelDigi> >(edm::DetSet<PixelDigi> const&, PixelGeomDetUnit const*, TrackerTopology const*, std::vector<short, std::allocator<short> > const&, edmNew::DetSetVector<SiPixelCluster>::FastFiller&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so
#10 0x00002b33061f949a in void SiPixelClusterProducer::run<edm::DetSetVector<PixelDigi> >(edm::DetSetVector<PixelDigi> const&, edm::ESHandle<TrackerGeometry> const&, edmNew::DetSetVector<SiPixelCluster>&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so
#11 0x00002b33061ec749 in SiPixelClusterProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so
#12 0x00002b32b32614c3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#13 0x00002b32b324a7df in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#14 0x00002b32b31a5c15 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#15 0x00002b32b31a5f0b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#16 0x00002b32b31a84f5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#17 0x00002b32b2f3c7d5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so
#18 0x00002b32b49c4b8c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x2b32b85d2900, this=0x2b32b85ee880) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
#19 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x2b32b85ee880) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
#20 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.cpp:168
#21 0x00002b32b3116908 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#22 0x00002b32b312163b in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#23 0x000000000040a266 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#24 0x00002b32b49b315b in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/arena.cpp:698
#25 0x000000000040b094 in main::{lambda()#1}::operator()() const ()
#26 0x000000000040971c in main ()

Current Modules:

Module: CAHitQuadrupletEDProducer:hltPixelTracksHitQuadrupletsPPOnAA (crashed)
Module: SiPixelClusterProducer:hltSiPixelClustersPPOnAA
Module: CaloTowersCreator:hltTowerMakerForAll
Module: HcalRawToDigi:hltHcalDigis

A fatal system signal has occurred: segmentation violation

@makortel
Copy link
Contributor

I'm guessing that failure is related to the issue discussed here

Looks the same, thanks. (and I'm pretty sure the culprit is the one diagnosed in #36804).

@AdrianoDee AdrianoDee mentioned this issue Jan 27, 2022
Merged
@missirol
Copy link
Contributor

missirol commented Jan 28, 2022 •
edited
Loading

Just for the record, the HLT-Validation tests also failed for IB CMSSW_12_3_X_2022-01-27-1100, again from a crash in CAHitQuadrupletEDProducer.

Interestingly, the same tests did not fail in IB CMSSW_12_3_X_2022-01-27-2300. Maybe this is related to the reproducibility issue in #34629 (or maybe I'm missing something).

@makortel
Copy link
Contributor

The occasional crash is caused by undefined behavior, so it is expected that only some jobs crash (since literally anything can happen). In IB runTheMatrix tests the rate of crashes is roughly 3/3.5k i.e. about 1 per mille (assuming all the workflows run tracking at some point).

@missirol
Copy link
Contributor

I see. Thanks for the clarification.

@mmusich
Copy link
Contributor

mmusich commented Feb 15, 2022

this is solved by #36819 right?
@cms-sw/reconstruction-l2 please sign and close.

@mmusich
Copy link
Contributor

mmusich commented Feb 28, 2022

kind ping on this issue (to be closed).

@perrotta
Copy link
Contributor

perrotta commented Apr 5, 2022

@dan131riley @cms-sw/reconstruction-l2 given #36819 (as commented above) I think this can be closed. Please reopen if you think that there is still some further work to do

@perrotta perrotta closed this as completed Apr 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
pending-signatures reconstruction-pending
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@makortel @dan131riley @perrotta @cmsbuild @mmusich @missirol