STK_Mesh failures in Nalu code base

[spdomin]

Bug Report

@trilinos/stk

@alanw0 and @jvo1012 - new clang core dumps - will know tomorrow if this extends to other platforms

Bad: commit 1f7e138
Author: Johnathan Vo jvo@sandia.gov
Good: commit 3c206dd

typical dump:

[Nalus:21801] [ 0] 0 libsystem_platform.dylib 0x00007fff72abc5fd _sigtramp + 29
[Nalus:21801] [ 1] 0 ??? 0x00007ff100001000 0x0 + 140673063849984
[Nalus:21801] [ 2] 0 naluX 0x0000000109ef34d4 _ZNK3stk4mesh8BulkData25shared_procs_intersectionERKNSt3__16vectorINS0_6EntityENS2_9allocatorIS4_EEEERNS3_IiNS5_IiEEEE + 100
[Nalus:21801] [ 3] 0 naluX 0x000000010a06dd41 _ZN3stk4mesh37fill_shared_entities_that_need_fixingERKNS0_8BulkDataE + 1009
[Nalus:21801] [ 4] 0 naluX 0x0000000109f0636b ZN3stk4mesh8BulkData33resolve_parallel_side_connectionsERNSt3__16vectorINS0_15SideSharingDataENS2_9allocatorIS4_EEEES8 + 59
[Nalus:21801] [ 5] 0 naluX 0x0000000109f06c0b _ZN3stk4mesh8BulkData48use_elem_elem_graph_to_determine_shared_entitiesERNSt3__16vectorINS0_6EntityENS2_9allocatorIS4_EEEE + 59
[Nalus:21801] [ 6] 0 naluX 0x0000000109f06e74 _ZN3stk4mesh8BulkData56fill_shared_entities_of_rank_while_updating_sharing_infoENS_8topology6rank_tERNSt3__16vectorINS0_6EntityENS4_9allocatorIS6_EEEE + 68
[Nalus:21801] [ 7] 0 naluX 0x0000000109f08cd8 _ZN3stk4mesh8BulkData32internal_resolve_parallel_createERKNSt3__16vectorINS_8topology6rank_tENS2_9allocatorIS5_EEEE + 504
[Nalus:21801] [ 8] 0 naluX 0x0000000109f088ab _ZN3stk4mesh8BulkData48internal_resolve_parallel_create_edges_and_facesEv + 331
[Nalus:21801] [ 9] 0 naluX 0x000000010a04201f _ZN3stk4mesh4impl16MeshModification55internal_modification_end_after_node_sharing_resolutionENS2_25modification_optimizationE + 47
[Nalus:21801] [10] 0 naluX 0x0000000109ebd061 _ZN3stk2io15StkMeshIoBroker22populate_mesh_sidesetsEb + 321
[Nalus:21801] [11] 0 naluX 0x00000001093d2b1a _ZN6sierra4nalu5Realm10initializeEv + 362
[Nalus:21801] [12] 0 naluX 0x00000001093f667a _ZN6sierra4nalu6Realms10initializeEv + 42
[Nalus:21801] [13] 0 naluX 0x00000001094144d2 _ZN6sierra4nalu10Simulation10initializeEv + 18
[Nalus:21801] [14] 0 naluX 0x000000010914e8ef main + 5727
[Nalus:21801] [15] 0 libdyld.dylib 0x00007fff728bfcc9 start + 1

Description

Steps to Reproduce

[jhux2]

@trilinos/stk

[alanw0]

@jhux2 is this also a exawind issue?
In any case, @spdomin it's been a while since I built nalu, I may need help with a recipe to build and reproduce this.

[spdomin]

@alanw0 and @jhux2, clang has been a struggle to test for the Nalu-based codes with Clang seemingly revolting. My guess is that we will learn more tomorrow morning and after Nalu and Nalu-Wind's regression test suite report. However, these are widespread on Nalu and I will guess Nalu-Wind will again track.

The bisect definitely points to this jvo push. We may want to hope for a gcc/intel fail since it will be easier to access at SNL.

I will report back tomorrow as the Clang state is a bit odd.

[spdomin]

@alanw0, otherwise, I have a full installation of Trilinos and other TPLs on gcc/intel and it's easy to build Nalu by pointing to this pre-installed path.

[alanw0]

@spdomin ok, let's see what tomorrow brings, and if possible I'll run this through totalview with a gcc build and see what's going on. jvo's stk snapshot was primarily to fix a few clang compiler errors, but it obviously also brought along other recent commits and I suspect something's wrong with one of those. Although it's puzzling because all of our commits go through gerrit which tests a full slate of sierra tests on 3 platforms... Anyway, we'll sort it out.

[spdomin]

@alanw0 - gcc seems fine after this commit:

NaluCFD/Nalu@43989dd

However, I thought that this change was incompatible with Clang. I probably need to look at EntitySorterBase to see if something is up there...

Perhaps in the morning, things will be more clear for me:)

[spdomin]

Yes, resolving the missing header in gcc results in the following for Clang:

In file included from /Users/naluIt/gitHubWork/nightlyBuildAndTest/Nalu/src/Realm.C:21:
In file included from /Users/naluIt/gitHubWork/nightlyBuildAndTest/Nalu/include/EntityExposedFaceSorter.h:14:
/Users/naluIt/gitHubWork/packages/install/Trilinos_nightly_release/include/stk_mesh/base/EntitySorterBase.hpp:42:7: error: redefinition of
'EntitySorterBase'
class EntitySorterBase
^
/Users/naluIt/gitHubWork/packages/install/Trilinos_nightly_release/include/stk_mesh/base/EntityLess.hpp:64:7: note: previous definition is
here
class EntitySorterBase
^
1 error generated.

[alanw0]

@spdomin ok that's a good clue. EntitySorterBase was moved from EntityLess.hpp into its own header. In sierra, EntitySorterBase no longer appears in EntityLess.hpp.
It looks like in github/trilinos, the master branch still has EntitySorterBase in EntityLess.hpp, while the develop branch has it in its own header EntitySorterBase.hpp.

[spdomin]

Interesting. How could that happen if a sync was just processed? There must be some subtlety in how these two repos communicate:)

Let me know when I can test something.

I think that when I remove the header include, the code builds on clang however segfaults. However let’s wait until we have a clean build Under clang before we worry about its testing behavior.

[alanw0]

It looks like develop was merged into master just after midnight, 12:07 this morning. So I guess it was just a window where the stk update had gone into develop and hadn't made it into master yet. But they appear to be in sync now.
So you do need your change where you include the new EntitySorterBase.hpp, but hopefully with that the build is good now.

So we should be ready to debug the runtime error you show at the top of this bug report, assuming that is still happening and wasn't caused by a bad build.

[spdomin]

I will check the Mac build now and report back. Debugging off this platform could be painful.

[jhux2]

is this also a exawind issue?

I think so: https://my.cdash.org/index.php?project=Nalu-Wind

@jrood-nrel

[alanw0]

@jhux2 there is a nalu-wind PR in progress to fix that build error.
@spdomin keep me posted

[spdomin]

Looks like the build is clean, however, clear failures noted. I thought the pattern may have been due to the face consolidated approach where we sort by exposed face topology, however, I see edge-based and non-consolidated element-based cases failing. As such, it is hard to see any pattern aside from the fact that its the same STK signature.

Do we have any ability to debug Clang off of the Mac? What about loading a Clang module off of my RHEL7 box? Is that an option?

1/1 Test #58: quad9HC ..........................***Failed    0.13 sec
[Nalus:86837] *** Process received signal ***
[Nalus:86837] Signal: Segmentation fault: 11 (11)
[Nalus:86837] Signal code: Address not mapped (1)
[Nalus:86837] Failing at address: 0x7
[Nalus:86837] [ 0] 0   libsystem_platform.dylib            0x00007fff72abc5fd _sigtramp + 29
[Nalus:86837] [ 1] 0   ???                                 0x0000000000000100 0x0 + 256
[Nalus:86837] [ 2] 0   naluX                               0x000000010acea354 _ZNK3stk4mesh8BulkData25shared_procs_intersectionERKNSt3__16vectorINS0_6EntityENS2_9allocatorIS4_EEEERNS3_IiNS5_IiEEEE + 100
[Nalus:86837] [ 3] 0   naluX                               0x000000010ae64bc1 _ZN3stk4mesh37fill_shared_entities_that_need_fixingERKNS0_8BulkDataE + 1009
[Nalus:86837] [ 4] 0   naluX                               0x000000010acfd1eb _ZN3stk4mesh8BulkData33resolve_parallel_side_connectionsERNSt3__16vectorINS0_15SideSharingDataENS2_9allocatorIS4_EEEES8_ + 59
[Nalus:86837] [ 5] 0   naluX                               0x000000010acfda8b _ZN3stk4mesh8BulkData48use_elem_elem_graph_to_determine_shared_entitiesERNSt3__16vectorINS0_6EntityENS2_9allocatorIS4_EEEE + 59
[Nalus:86837] [ 6] 0   naluX                               0x000000010acfdcf4 _ZN3stk4mesh8BulkData56fill_shared_entities_of_rank_while_updating_sharing_infoENS_8topology6rank_tERNSt3__16vectorINS0_6EntityENS4_9allocatorIS6_EEEE + 68
[Nalus:86837] [ 7] 0   naluX                               0x000000010acffb58 _ZN3stk4mesh8BulkData32internal_resolve_parallel_createERKNSt3__16vectorINS_8topology6rank_tENS2_9allocatorIS5_EEEE + 504
[Nalus:86837] [ 8] 0   naluX                               0x000000010acff72b _ZN3stk4mesh8BulkData48internal_resolve_parallel_create_edges_and_facesEv + 331
[Nalus:86837] [ 9] 0   naluX                               0x000000010ae38e9f _ZN3stk4mesh4impl16MeshModification55internal_modification_end_after_node_sharing_resolutionENS2_25modification_optimizationE + 47
[Nalus:86837] [10] 0   naluX                               0x000000010acb3ee1 _ZN3stk2io15StkMeshIoBroker22populate_mesh_sidesetsEb + 321
[Nalus:86837] [11] 0   naluX                               0x000000010a1c95ea _ZN6sierra4nalu5Realm10initializeEv + 362
[Nalus:86837] [12] 0   naluX                               0x000000010a1ed14a _ZN6sierra4nalu6Realms10initializeEv + 42
[Nalus:86837] [13] 0   naluX                               0x000000010a20afa2 _ZN6sierra4nalu10Simulation10initializeEv + 18
[Nalus:86837] [14] 0   naluX                               0x0000000109f453bf main + 5727
[Nalus:86837] [15] 0   libdyld.dylib                       0x00007fff728bfcc9 start + 1
[Nalus:86837] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 86837 on node Nalus exited on signal 11 (Segmentation fault: 11).
--------------------------------------------------------------------------

2/2 Test #38: milestoneRunConsolidated .........***Failed    0.30 sec

IOSS: Using decomposition method 'RCB' for 34,215 elements on 4 processors.
[Nalus:86881] *** Process received signal ***
[Nalus:86881] Signal: Segmentation fault: 11 (11)
[Nalus:86881] Signal code: Address not mapped (1)
[Nalus:86881] Failing at address: 0x8
[Nalus:86881] [ 0] 0   libsystem_platform.dylib            0x00007fff72abc5fd _sigtramp + 29
[Nalus:86881] [ 1] 0   ???                                 0x0000000000001000 0x0 + 4096
[Nalus:86881] [ 2] 0   naluX                               0x000000010c19e354 _ZNK3stk4mesh8BulkData25shared_procs_intersectionERKNSt3__16vectorINS0_6EntityENS2_9allocatorIS4_EEEERNS3_IiNS5_IiEEEE + 100
[Nalus:86881] [ 3] 0   naluX                               0x000000010c318bc1 _ZN3stk4mesh37fill_shared_entities_that_need_fixingERKNS0_8BulkDataE + 1009
[Nalus:86881] [ 4] 0   naluX                               0x000000010c1b11eb _ZN3stk4mesh8BulkData33resolve_parallel_side_connectionsERNSt3__16vectorINS0_15SideSharingDataENS2_9allocatorIS4_EEEES8_ + 59
[Nalus:86881] [ 5] 0   naluX                               0x000000010c1b1a8b _ZN3stk4mesh8BulkData48use_elem_elem_graph_to_determine_shared_entitiesERNSt3__16vectorINS0_6EntityENS2_9allocatorIS4_EEEE + 59
[Nalus:86881] [ 6] 0   naluX                               0x000000010c1b1cf4 _ZN3stk4mesh8BulkData56fill_shared_entities_of_rank_while_updating_sharing_infoENS_8topology6rank_tERNSt3__16vectorINS0_6EntityENS4_9allocatorIS6_EEEE + 68
[Nalus:86881] [ 7] 0   naluX                               0x000000010c1b3b58 _ZN3stk4mesh8BulkData32internal_resolve_parallel_createERKNSt3__16vectorINS_8topology6rank_tENS2_9allocatorIS5_EEEE + 504
[Nalus:86881] [ 8] 0   naluX                               0x000000010c1b372b _ZN3stk4mesh8BulkData48internal_resolve_parallel_create_edges_and_facesEv + 331
[Nalus:86881] [ 9] 0   naluX                               0x000000010c2ece9f _ZN3stk4mesh4impl16MeshModification55internal_modification_end_after_node_sharing_resolutionENS2_25modification_optimizationE + 47
[Nalus:86881] [10] 0   naluX                               0x000000010c167ee1 _ZN3stk2io15StkMeshIoBroker22populate_mesh_sidesetsEb + 321
[Nalus:86881] [11] 0   naluX                               0x000000010b67d5ea _ZN6sierra4nalu5Realm10initializeEv + 362
[Nalus:86881] [12] 0   naluX                               0x000000010b6a114a _ZN6sierra4nalu6Realms10initializeEv + 42
[Nalus:86881] [13] 0   naluX                               0x000000010b6befa2 _ZN6sierra4nalu10Simulation10initializeEv + 18
[Nalus:86881] [14] 0   naluX                               0x000000010b3f93bf main + 5727
[Nalus:86881] [15] 0   libdyld.dylib                       0x00007fff728bfcc9 start + 1
[Nalus:86881] [16] 0   ???                                 0x0000000000000005 0x0 + 5
[Nalus:86881] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 1 with PID 86881 on node Nalus exited on signal 11 (Segmentation fault: 11).
--------------------------------------------------------------------------

0% tests passed, 2 tests failed out of 2

[spdomin]

Also, I am using a rather new clang (11) while I see Nalu-Wind is testing with 9-ish.

[alanw0]

@spdomin is this run-time error only happening with clang? Or is it also happening with gcc etc?

[spdomin]

Only clang. The gcc 8.3 was clean.

[alanw0]

ok, I think my first move will be to do a sierra stk build with clang. We have a number of clang modules for sierra, but no dashboard lines for it. If I can reproduce there, that's the easiest for me. I'll let you know asap what I come up with.

[spdomin]

I will check intel 19 now. I generally only run gcc and mac nightly. However it’s no prob to check intel.

Sierra looks like it may use clang 10. I guess I would use the highest one.

[spdomin]

Looks like Intel 19 is also clean.

There seems to be a secrete recipe to map between Mac clang version and official clang version.

Perhaps someone could fire off a Nalu-Wind Darwin build/test to see if (after the latest header PR), the suite is clean?

[spdomin]

I am also trying to update open mpi on my Mac platform.

[spdomin]

Too bad:(

I upgraded my Mac build to the latest open MPI (4.0.4) and still see these core dumps. Note that if I run the failing test in serial, it runs and passes.

I am also looking into a similar Clang version on my linux RHEL7 system. Here, it seems that the conversions are Mac/Clang 11.0.3 --> Clang 9.0

We may want to wait to see what Nalu-Wind does, although the fact Nalu-proper is failing on this platform suggests something off. As far as i can tell, my Mac environment is sane and the new change definitely caused the failures.

Perhaps I should try running an STK unit test on my Trilinos build? How might I do that?

[tasmith4]

@spdomin I believe the stk_mesh unit tests get built under packages/stk/stk_unit_tests/stk_mesh in your Trilinos build directory. You might need to have explicitly enabled tests on your CMake configure line.

[alanw0]

Can you run totalview on that mac machine? If you can get a debug build and run totalview, maybe I could log in and try to see what's going on. I agree it seems like a code problem, I'm just getting low on ideas for reproducing it..
There should be unit-tests built for several stk packages if you have tests enabled for your trilinos build. in your build directory try something like 'find . -name *exe' to find the unit test executables. If stk-io unit-tests are built, that could be promising since your stack trace goes down through stk-io.

[spdomin]

I do not have an effective debugger on my Mac system. I circled this topic with Anthony for a while, however, was never able to obtain a viable TV option. This fact makes it very hard to debug. I may have ddd:)

I will work on the stk unit tests. I seem to recall I had to turn these off to avoid adding more dependencies to my build, e.g., loadbal.

[spdomin]

Final report for the day: My RHEL7-based Clang 9.0, open mpi 4.0.3 also behaves well. Thus far, gcc 8.3.0 + open mpi 4.0.3, Intel 19 + open mpi 4.0.3, and clang 9.0 + open mpi 4.0.3 all work.

It is only the Darwin build using Clang 11.0.3 + a few flavors of open mpi (tested the most recent 4.0.4 version) that are failing.

I plan on trying to run the STK unit test suite tomorrow PM and reporting back. In the meantime, we will see what the nightly Nalu-wind tests results show.

[alanw0]

ok, tomorrow is another day, hopefully we can get to the bottom of this...

[jhux2]

@spdomin valgrind is supposed to run on MacOS, maybe that would be helpful? Maybe old fashioned gdb?

[nmhamster]

@spdomin / @jhux2 - you should have lldb installed by Apple's SDK that provides gdb like debugging. It isn't the world's greatest but with a bit of playing around it will do back tracing etc for you and should give us some kind of source code location. If I can help at all, let me know.

[alanw0]

It looks like nalu-wind tests are failing on a linux clang platform with address sanitizer. I'm now trying a build with address sanitizer, hopefully that will reproduce this.

[spdomin]

@alanw0, yes:

#2 0x7fdba6312265 in stk::mesh::BulkData::shared_procs_intersection(std::__1::vector<stk::mesh::Entity, std::__1::allocator<stk::mesh::Entity> > const&, std::__1::vector<int, std::__1::allocator<int> >&) const /projects/ecp/exawind/nalu-wind-testing/spack/var/spack/stage/trilinos-develop-imgt6ae2bqw2752jubbuiu6epnznpsbz/src/packages/stk/stk_mesh/stk_mesh/base/BulkData.cpp:1731:7
@nmhamster , thanks for the suggestion. I have always resisted using lldb, however, (for fun/pain) may fire it up. In my comments above, I thought I said that "ddd" is an option? I have used this before on linux-based platforms. Here, at least a graphical interface exists.

The Darwin platform for Nalu-Wind worked fine. Argh - at least we have something common here.

[alanw0]

I'm currently beefing up the unit-test for that function a little more, and also trying an address sanitizer build.
One thing we know: this function was changed recently, and it appears to be the site of the error. So today, I will either find/fix this error, or revert that change. It was a good change, an optimization that improved speed and also greatly reduce usage of new/delete. However, it seems clear that it introduced a subtle memory error in some scenario yet to be understood.

[alanw0]

I have a pull-request which attempts to fix this issue. #7836
FYI @sayerhs since it seems to be the same issue in nalu-wind's linux-clang failures.

[spdomin]

Looks promising... After cloning Alan's fork and checking out his patch, it looks like the selective testing (spot checking here and there) resulted in the previous failing tests to pass.

I suppose all this means is that the selective revert based on the call stack resulted in a sane build. As you (the STK team) debug this clang issue, let me know if I can help.

naluIt$ ctest -R milestone
Test project /Users/naluIt/gitHubWork/nightlyBuildAndTest/Nalu/build
    Start 37: milestoneRun
1/2 Test #37: milestoneRun .....................   Passed   45.72 sec
    Start 38: milestoneRunConsolidated
2/2 Test #38: milestoneRunConsolidated .........   Passed   47.48 sec

[alanw0]

@spdomin thanks for checking that. Yes I will keep working to reproduce the problem with a clang build in the sierra system, because I would like to get that code un-reverted if possible, since it was a nice optimization.

[spdomin]

@spdomin thanks for checking that. Yes I will keep working to reproduce the problem with a clang build in the sierra system, because I would like to get that code un-reverted if possible, since it was a nice optimization.

Agreed. I will work on that debug-option that Si suggested. There is still the complexity of getting that Mac platform to your eyes.... I think that the local Clang replication is our best hope. If we find the exact details of that platform, I can replicate it locally.

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

[github-actions]

This issue was closed due to inactivity for 395 days.