segfaults in ThreadSafeLogMessageLoggerScribe on slc7_ppc64le_gcc9 #33636
Labels
Comments
|
A new Issue was created by @dan131riley Dan Riley. @Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks. cms-bot commands are listed here |
|
Also a weird segfault in |
|
assign core |
|
New categories assigned: core @Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks |
|
These crashes are in (at least) wf 31434.0, 34834.999, 35034.0, 35434.0, 36234.0, and 36634.0. All three are Phase2C11* Extended2026* variants with HGCal. They mostly track back to MessageLogger messages in cmssw/Geometry/HGCalCommonData/src/HGCalDDDConstants.cc Lines 1601 to 1603 in f6e2225 but also a few from or cmssw/DQM/SiTrackerPhase2/plugins/Phase2TrackerMonitorDigi.cc Lines 284 to 285 in 0c38ca4 I don't see any obvious reason these would crash, I guess try valgrind? |
|
@dan131riley it looks to me like the HGCalDDDConstants was meant to be behind a |
|
Having debug lines would help in tracking down the MessageLogger problem. |
|
Another one in |
|
Another in |
|
@smuzaffar @mrodozov would it be possible to build the PPC IB with debug symbols? Getting line numbers for the crash would be super useful. |
|
I'll check if I can bring the _DBG_ IB for PPC |
|
I'm adding a hint for myself here to not forget the new relvals failures, apart from the Geom ones may be failing because of this |
In general, for non-production architectures, would it make sense to always build debug symbols but not necessarily with optimization disabled, i.e. just |
|
There might be a problem when building in debug all "experimental" IBs because we use shared resources for that and it might take longer for testing. But we can try few times and check if it actually does, if it doesn't maybe yes. For PPC this may fly under the radar but I doubt it on Arm machines which are slow even now |
My suggestion is to try just generating debug symbol, |
|
We can try that yes. The DEBUG PPC IB will appear with -1700 hour some time soon |
|
@mrodozov so we had another seg fault in the 2300 build on ppc but there were no additional debug information. Was the debug only applied to one specific build? |
|
to the one specific build only in 1700, yes. the one in 2300 was with the existing compiler options (no debug info yet) |
|
@mrodozov The 1700 build failed, it was compiled with "-g -O0" and we apparently still have some modules that won't build correctly with optimization off. We should try to fix those, but in the meantime is it possible to do what @Dr15Jones suggested, add "-g" but leave the normal optimization flags? |
|
I'm searching where are the options for the _DBG_ IBs so I can turn this optimization to 3 from 0 |
|
|
done. github is not very useful for searching basic things 😃 (I did search for _DBG_ rather then DBG only tho) |
|
started an IB for -2100 (and the right date this time) |
|
Bad/good/bad news story. Bad: even with optimization on, there's still a bunch of stuff that doesn't compile with EDM_ML_DEBUG defined, so there's lots in the latest build that still didn't compile. We need a campaign to make DBG builds work. Good: enough worked that we did get a bunch of stack traces, partly because EDM_ML_DEBUG turned on a bunch more logging. Representative stack trace below, no conclusions yet. Bad: none of the more elusive |
Yep, so it's probably a likely source of the crash because it gets executed a lot more than intended, similar to how turning on EDM_ML_DEBUG produces a much higher likelihood of crashes (and compilation failures). |
|
I started fixing the obvious stuff although I'm not sure with what rate I fix things and new failures are put for example in L1Trigger 😃 |
|
See #33734 to track issues with DBG build |
|
the framework missing lib should be fixed with cms-sw/cmsdist#6906 (tested on ppc to make sure this time) |
|
The place where the MessageLogger and where |
|
So looking at the failure in the _DBG build, the method in TBB that is failing is the following (from: /cvmfs/cms-ib.cern.ch/nweek-02680/slc7_amd64_gcc900/external/tbb/v2021.2.0/include/oneapi/tbb/detail/_concurrent_queue_base.h ) 175 bool pop( void* dst, ticket_type k, queue_rep_type& base ) {
176 k &= -queue_rep_type::n_queue;
177 if (head_counter.load(std::memory_order_relaxed) != k) spin_wait_until_eq(head_counter, k);
178 call_itt_notify(acquired, &head_counter);
179 if (tail_counter.load(std::memory_order_relaxed) == k) spin_wait_while_eq(tail_counter, k);
180 call_itt_notify(acquired, &tail_counter);
181 padded_page *p = head_page.load(std::memory_order_acquire);
182 __TBB_ASSERT( p, nullptr );
183 size_type index = modulo_power_of_two( k/queue_rep_type::n_queue, items_per_page );
184 bool success = false;
185 {
186 page_allocator_type page_allocator(base.get_allocator());
187 micro_queue_pop_finalizer<self_type, value_type, page_allocator_type> finalizer(*this, page_allocator,
188 k + queue_rep_type::n_queue, index == items_per_page - 1 ? p : nullptr );
189 if (p->mask.load(std::memory_order_relaxed) & (std::uintptr_t(1) << index)) {
190 success = true;
191 assign_and_destroy_item( dst, *p, index );
192 } else {
193 --base.n_invalid_entries;
194 }
195 }
196 return success;
197 }It fails at line 189 with a segmentation fault. The most likely reason is |
|
We got a segfault in |
|
Can we try cherry picking oneapi-src/oneTBB#435? Use after scope on the allocator is a plausible explanation for our crashes. |
I'd better suggest trying the current |
|
sure I'll get it. |
|
Cherry-picking the commits from the oneTBB PR 435 didn't compile on top of our 2021.2.0 (breaking changes and change of namespace name which I didn't go to try and understand, and eventually stitch). |
mrodozov
added a commit
to cms-sw/cmsdist
that referenced
this issue
Jun 17, 2021
See cms-sw/cmssw#33636 (comment) we don't have tbb external and we don't need it for this temp test on PPC I've used my fork
|
We built an IB CMSSW_12_0/2021-06-19-1100 for slc7_ppc64le_gcc9 that doesn't have "the usual" MessageLogger failures |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@gartung following the update to TBB 2021.2.0 in #33474 and cms-sw/cmsdist#6792 we're seeing what looks like a race condition in the message logger on ppc:
The text was updated successfully, but these errors were encountered: