[2023-10-10T14:22:18] Walked 538399843 items in 6806.207 secs (79104.243 items/sec) ... [2023-10-10T14:22:19] Walked 538399843 items in 6806.941 seconds (79095.714 items/sec) 3: local_size=8207451 global_size=538399843 chars=794 14: local_size=8504979 global_size=538399843 chars=794 30: local_size=7494214 global_size=538399843 chars=794 33: local_size=11602229 global_size=538399843 chars=794 25: local_size=8210408 global_size=538399843 chars=794 22: local_size=10401127 global_size=538399843 chars=794 13: local_size=8003219 global_size=538399843 chars=794 61: local_size=8038447 global_size=538399843 chars=794 1: local_size=10044309 global_size=538399843 chars=794 19: local_size=7902603 global_size=538399843 chars=794 40: local_size=7381952 global_size=538399843 chars=794 55: local_size=8165772 global_size=538399843 chars=794 31: local_size=7205289 global_size=538399843 chars=794 39: local_size=7854120 global_size=538399843 chars=794 2: local_size=8979569 global_size=538399843 chars=794 10: local_size=10092005 global_size=538399843 chars=794 54: local_size=8103485 global_size=538399843 chars=794 4: local_size=7287968 global_size=538399843 chars=794 21: local_size=6638274 global_size=538399843 chars=794 26: local_size=11674197 global_size=538399843 chars=794 36: local_size=11932060 global_size=538399843 chars=794 43: local_size=7497463 global_size=538399843 chars=794 16: local_size=9205030 global_size=538399843 chars=794 24: local_size=9055343 global_size=538399843 chars=794 32: local_size=5871898 global_size=538399843 chars=794 27: local_size=8314818 global_size=538399843 chars=794 60: local_size=8488139 global_size=538399843 chars=794 37: local_size=7823217 global_size=538399843 chars=794 29: local_size=6995827 global_size=538399843 chars=794 53: local_size=8258029 global_size=538399843 chars=794 38: local_size=7362017 global_size=538399843 chars=794 15: local_size=11281268 global_size=538399843 chars=794 63: local_size=11549531 global_size=538399843 chars=794 59: local_size=12458992 global_size=538399843 chars=794 17: local_size=11268171 global_size=538399843 chars=794 47: local_size=8095051 global_size=538399843 chars=794 6: local_size=8025969 global_size=538399843 chars=794 9: local_size=8671116 global_size=538399843 chars=794 48: local_size=7413615 global_size=538399843 chars=794 11: local_size=9190970 global_size=538399843 chars=794 51: local_size=12067788 global_size=538399843 chars=794 28: local_size=8100658 global_size=538399843 chars=794 20: local_size=6765291 global_size=538399843 chars=794 23: local_size=7985656 global_size=538399843 chars=794 5: local_size=7132828 global_size=538399843 chars=794 41: local_size=6245288 global_size=538399843 chars=794 57: local_size=7579222 global_size=538399843 chars=794 8: local_size=7272524 global_size=538399843 chars=794 7: local_size=7584229 global_size=538399843 chars=794 50: local_size=7556251 global_size=538399843 chars=794 12: local_size=7582701 global_size=538399843 chars=794 42: local_size=7790335 global_size=538399843 chars=794 49: local_size=7178817 global_size=538399843 chars=794 56: local_size=7484803 global_size=538399843 chars=794 35: local_size=6751806 global_size=538399843 chars=794 52: local_size=6436153 global_size=538399843 chars=794 34: local_size=7060049 global_size=538399843 chars=794 46: local_size=8657357 global_size=538399843 chars=794 45: local_size=8128008 global_size=538399843 chars=794 44: local_size=7135070 global_size=538399843 chars=794 58: local_size=6393166 global_size=538399843 chars=794 62: local_size=8761387 global_size=538399843 chars=794 18: local_size=8048416 global_size=538399843 chars=794 0: local_size=12151899 global_size=538399843 chars=794 41: key_extent=794, keysat_extent=1676, bufsize=1877168096 exp=1877168096 32: key_extent=794, keysat_extent=1676, bufsize=1251366456 exp=1251366456 52: key_extent=794, keysat_extent=1676, bufsize=-2097909460 exp=-2097909460 58: key_extent=794, keysat_extent=1676, bufsize=2125011624 exp=2125011624 21: key_extent=794, keysat_extent=1676, bufsize=-1759154664 exp=-1759154664 35: key_extent=794, keysat_extent=1676, bufsize=-1568875032 exp=-1568875032 20: key_extent=794, keysat_extent=1676, bufsize=-1546274172 exp=-1546274172 49: key_extent=794, keysat_extent=1676, bufsize=-853204596 exp=-853204596 29: key_extent=794, keysat_extent=1676, bufsize=-1159895836 exp=-1159895836 44: key_extent=794, keysat_extent=1676, bufsize=-926524568 exp=-926524568 40: key_extent=794, keysat_extent=1676, bufsize=-512750336 exp=-512750336 34: key_extent=794, keysat_extent=1676, bufsize=-1052259764 exp=-1052259764 31: key_extent=794, keysat_extent=1676, bufsize=-808837524 exp=-808837524 56: key_extent=794, keysat_extent=1676, bufsize=-340372060 exp=-340372060 5: key_extent=794, keysat_extent=1676, bufsize=-930282160 exp=-930282160 4: key_extent=794, keysat_extent=1676, bufsize=-670267520 exp=-670267520 50: key_extent=794, keysat_extent=1676, bufsize=-220625212 exp=-220625212 43: key_extent=794, keysat_extent=1676, bufsize=-319153900 exp=-319153900 8: key_extent=794, keysat_extent=1676, bufsize=-696151664 exp=-696151664 48: key_extent=794, keysat_extent=1676, bufsize=-459683148 exp=-459683148 12: key_extent=794, keysat_extent=1676, bufsize=-176295012 exp=-176295012 30: key_extent=794, keysat_extent=1676, bufsize=-324599224 exp=-324599224 57: key_extent=794, keysat_extent=1676, bufsize=-182125816 exp=-182125816 38: key_extent=794, keysat_extent=1676, bufsize=-546161396 exp=-546161396 7: key_extent=794, keysat_extent=1676, bufsize=-173734084 exp=-173734084 42: key_extent=794, keysat_extent=1676, bufsize=171699572 exp=171699572 13: key_extent=794, keysat_extent=1676, bufsize=528493156 exp=528493156 23: key_extent=794, keysat_extent=1676, bufsize=499057568 exp=499057568 39: key_extent=794, keysat_extent=1676, bufsize=278603232 exp=278603232 19: key_extent=794, keysat_extent=1676, bufsize=359860740 exp=359860740 54: key_extent=794, keysat_extent=1676, bufsize=696538972 exp=696538972 28: key_extent=794, keysat_extent=1676, bufsize=691800920 exp=691800920 47: key_extent=794, keysat_extent=1676, bufsize=682403588 exp=682403588 61: key_extent=794, keysat_extent=1676, bufsize=587535284 exp=587535284 37: key_extent=794, keysat_extent=1676, bufsize=226809804 exp=226809804 45: key_extent=794, keysat_extent=1676, bufsize=737639520 exp=737639520 3: key_extent=794, keysat_extent=1676, bufsize=870785988 exp=870785988 18: key_extent=794, keysat_extent=1676, bufsize=604243328 exp=604243328 6: key_extent=794, keysat_extent=1676, bufsize=566622156 exp=566622156 55: key_extent=794, keysat_extent=1676, bufsize=800931984 exp=800931984 25: key_extent=794, keysat_extent=1676, bufsize=875741920 exp=875741920 60: key_extent=794, keysat_extent=1676, bufsize=1341219076 exp=1341219076 53: key_extent=794, keysat_extent=1676, bufsize=955554716 exp=955554716 27: key_extent=794, keysat_extent=1676, bufsize=1050733080 exp=1050733080 14: key_extent=794, keysat_extent=1676, bufsize=1369442916 exp=1369442916 62: key_extent=794, keysat_extent=1676, bufsize=1799182724 exp=1799182724 9: key_extent=794, keysat_extent=1676, bufsize=1647888528 exp=1647888528 46: key_extent=794, keysat_extent=1676, bufsize=1624828444 exp=1624828444 24: key_extent=794, keysat_extent=1676, bufsize=-2003114316 exp=-2003114316 16: key_extent=794, keysat_extent=1676, bufsize=-1752238904 exp=-1752238904 11: key_extent=794, keysat_extent=1676, bufsize=-1775803464 exp=-1775803464 2: key_extent=794, keysat_extent=1676, bufsize=-2130111540 exp=-2130111540 22: key_extent=794, keysat_extent=1676, bufsize=252419668 exp=252419668 1: key_extent=794, keysat_extent=1676, bufsize=-345607300 exp=-345607300 10: key_extent=794, keysat_extent=1676, bufsize=-265668804 exp=-265668804 15: key_extent=794, keysat_extent=1676, bufsize=1727535984 exp=1727535984 33: key_extent=794, keysat_extent=1676, bufsize=-2029500676 exp=-2029500676 36: key_extent=794, keysat_extent=1676, bufsize=-1476703920 exp=-1476703920 17: key_extent=794, keysat_extent=1676, bufsize=1705585412 exp=1705585412 63: key_extent=794, keysat_extent=1676, bufsize=-2117822524 exp=-2117822524 26: key_extent=794, keysat_extent=1676, bufsize=-1908882308 exp=-1908882308 51: key_extent=794, keysat_extent=1676, bufsize=-1249223792 exp=-1249223792 59: key_extent=794, keysat_extent=1676, bufsize=-593565888 exp=-593565888 0: key_extent=794, keysat_extent=1676, bufsize=-1108253756 exp=-1108253756 [luna-0078:3879655:0:3879655] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x7fdf511f3000) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) ==== backtrace (tid:3879655) ==== 0 0x0000000000043090 killpg() ???:0 1 0x000000000018b9c9 __nss_database_lookup() ???:0 2 0x0000000000061012 dtcmp_merge_local_2way_memcpy() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_merge_2way.c:52 3 0x00000000000639f0 dtcmp_sort_local_mergesort_scratch() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:43 4 0x000000000006399c dtcmp_sort_local_mergesort_scratch() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:41 5 0x00000000000639c1 dtcmp_sort_local_mergesort_scratch() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:42 6 0x0000000000063b1f DTCMP_Sort_local_mergesort() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:77 7 0x000000000005b858 DTCMP_Sort_local() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1129 8 0x0000000000067828 DTCMP_Sortv_cheng_lwgrp() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sortv_cheng.c:759 9 0x00000000000679f9 DTCMP_Sortv_cheng() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sortv_cheng.c:809 10 0x000000000005bbf5 DTCMP_Sortv() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1291 11 0x000000000005bdea DTCMP_Sortz() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1365 12 0x00000000000408d5 sort_files_stat() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/common/mfu_flist_sort.c:545 13 0x0000000000040b2f mfu_flist_sort() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/common/mfu_flist_sort.c:612 14 0x0000000000003e09 main() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/dwalk/dwalk.c:670 15 0x0000000000024083 __libc_start_main() ???:0 16 0x00000000000026ee _start() ???:0 ================================= [luna-0078:3879655] *** Process received signal *** [luna-0078:3879655] Signal: Segmentation fault (11) [luna-0078:3879655] Signal code: (-6) [luna-0078:3879655] Failing at address: 0x3b32e7 [luna-0078:3879655] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7fdf7ee80090] [luna-0078:3879655] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x18b9c9)[0x7fdf7efc89c9] [luna-0078:3879655] [ 2] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(dtcmp_merge_local_2way_memcpy+0x158)[0x7fdf7f321012] [luna-0078:3879655] [ 3] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x639f0)[0x7fdf7f3239f0] [luna-0078:3879655] [ 4] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x6399c)[0x7fdf7f32399c] [luna-0078:3879655] [ 5] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x639c1)[0x7fdf7f3239c1] [luna-0078:3879655] [ 6] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sort_local_mergesort+0xf0)[0x7fdf7f323b1f] [luna-0078:3879655] [ 7] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sort_local+0xf8)[0x7fdf7f31b858] [luna-0078:3879655] [ 8] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv_cheng_lwgrp+0x1a1)[0x7fdf7f327828] [luna-0078:3879655] [ 9] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv_cheng+0x7e)[0x7fdf7f3279f9] [luna-0078:3879655] [10] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv+0x1c2)[0x7fdf7f31bbf5] [luna-0078:3879655] [11] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortz+0x1db)[0x7fdf7f31bdea] [luna-0078:3879655] [12] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x408d5)[0x7fdf7f3008d5] [luna-0078:3879655] [13] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(mfu_flist_sort+0xa6)[0x7fdf7f300b2f] [luna-0078:3879655] [14] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/mpifileutils/src/dwalk/dwalk(main+0xb8f)[0x55f5ed5a9e09] [luna-0078:3879655] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fdf7ee61083] [luna-0078:3879655] [16] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/mpifileutils/src/dwalk/dwalk(_start+0x2e)[0x55f5ed5a86ee] [luna-0078:3879655] *** End of error message *** [luna-0059:3835785:0:3835785] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x7fa9bc2a1000) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) ==== backtrace (tid:3835785) ==== 0 0x0000000000043090 killpg() ???:0 1 0x000000000018b9d3 __nss_database_lookup() ???:0 2 0x0000000000060fe2 dtcmp_merge_local_2way_memcpy() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_merge_2way.c:49 3 0x00000000000639f0 dtcmp_sort_local_mergesort_scratch() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:43 4 0x000000000006399c dtcmp_sort_local_mergesort_scratch() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:41 5 0x000000000006399c dtcmp_sort_local_mergesort_scratch() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:41 6 0x0000000000063b1f DTCMP_Sort_local_mergesort() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:77 7 0x000000000005b858 DTCMP_Sort_local() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1129 8 0x0000000000067828 DTCMP_Sortv_cheng_lwgrp() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sortv_cheng.c:759 9 0x00000000000679f9 DTCMP_Sortv_cheng() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sortv_cheng.c:809 10 0x000000000005bbf5 DTCMP_Sortv() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1291 11 0x000000000005bdea DTCMP_Sortz() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1365 12 0x00000000000408d5 sort_files_stat() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/common/mfu_flist_sort.c:545 13 0x0000000000040b2f mfu_flist_sort() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/common/mfu_flist_sort.c:612 14 0x0000000000003e09 main() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/dwalk/dwalk.c:670 15 0x0000000000024083 __libc_start_main() ???:0 16 0x00000000000026ee _start() ???:0 ================================= [luna-0059:3835785] *** Process received signal *** [luna-0059:3835785] Signal: Segmentation fault (11) [luna-0059:3835785] Signal code: (-6) [luna-0059:3835785] Failing at address: 0x3a8789 [luna-0059:3835785] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7fa9ece72090] [luna-0059:3835785] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x18b9d3)[0x7fa9ecfba9d3] [luna-0059:3835785] [ 2] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(dtcmp_merge_local_2way_memcpy+0x128)[0x7fa9ed312fe2] [luna-0059:3835785] [ 3] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x639f0)[0x7fa9ed3159f0] [luna-0059:3835785] [ 4] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x6399c)[0x7fa9ed31599c] [luna-0059:3835785] [ 5] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x6399c)[0x7fa9ed31599c] [luna-0059:3835785] [ 6] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sort_local_mergesort+0xf0)[0x7fa9ed315b1f] [luna-0059:3835785] [ 7] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sort_local+0xf8)[0x7fa9ed30d858] [luna-0059:3835785] [ 8] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv_cheng_lwgrp+0x1a1)[0x7fa9ed319828] [luna-0059:3835785] [ 9] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv_cheng+0x7e)[0x7fa9ed3199f9] [luna-0059:3835785] [10] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv+0x1c2)[0x7fa9ed30dbf5] [luna-0059:3835785] [11] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortz+0x1db)[0x7fa9ed30ddea] [luna-0059:3835785] [12] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x408d5)[0x7fa9ed2f28d5] [luna-0059:3835785] [13] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(mfu_flist_sort+0xa6)[0x7fa9ed2f2b2f] [luna-0059:3835785] [14] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/mpifileutils/src/dwalk/dwalk(main+0xb8f)[0x556444041e09] [luna-0059:3835785] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fa9ece53083] [luna-0059:3835785] [16] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/mpifileutils/src/dwalk/dwalk(_start+0x2e)[0x5564440406ee] [luna-0059:3835785] *** End of error message *** [luna-0059:3835769:0:3835769] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x7f50efd41000) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) ==== backtrace (tid:3835769) ==== 0 0x0000000000043090 killpg() ???:0 1 0x000000000018b9c5 __nss_database_lookup() ???:0 2 0x0000000000061012 dtcmp_merge_local_2way_memcpy() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_merge_2way.c:52 3 0x00000000000639f0 dtcmp_sort_local_mergesort_scratch() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:43 4 0x00000000000639c1 dtcmp_sort_local_mergesort_scratch() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:42 5 0x000000000006399c dtcmp_sort_local_mergesort_scratch() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:41 6 0x0000000000063b1f DTCMP_Sort_local_mergesort() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:77 7 0x000000000005b858 DTCMP_Sort_local() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1129 8 0x0000000000067828 DTCMP_Sortv_cheng_lwgrp() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sortv_cheng.c:759 9 0x00000000000679f9 DTCMP_Sortv_cheng() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sortv_cheng.c:809 10 0x000000000005bbf5 DTCMP_Sortv() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1291 11 0x000000000005bdea DTCMP_Sortz() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1365 12 0x00000000000408d5 sort_files_stat() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/common/mfu_flist_sort.c:545 13 0x0000000000040b2f mfu_flist_sort() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/common/mfu_flist_sort.c:612 14 0x0000000000003e09 main() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/dwalk/dwalk.c:670 15 0x0000000000024083 __libc_start_main() ???:0 16 0x00000000000026ee _start() ???:0 ================================= [luna-0059:3835769] *** Process received signal *** [luna-0059:3835769] Signal: Segmentation fault (11) [luna-0059:3835769] Signal code: (-6) [luna-0059:3835769] Failing at address: 0x3a8779 [luna-0059:3835769] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7f5126a82090] [luna-0059:3835769] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x18b9c5)[0x7f5126bca9c5] [luna-0059:3835769] [ 2] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(dtcmp_merge_local_2way_memcpy+0x158)[0x7f5126f23012] [luna-0059:3835769] [ 3] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x639f0)[0x7f5126f259f0] [luna-0059:3835769] [ 4] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x639c1)[0x7f5126f259c1] [luna-0059:3835769] [ 5] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x6399c)[0x7f5126f2599c] [luna-0059:3835769] [ 6] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sort_local_mergesort+0xf0)[0x7f5126f25b1f] [luna-0059:3835769] [ 7] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sort_local+0xf8)[0x7f5126f1d858] [luna-0059:3835769] [ 8] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv_cheng_lwgrp+0x1a1)[0x7f5126f29828] [luna-0059:3835769] [ 9] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv_cheng+0x7e)[0x7f5126f299f9] [luna-0059:3835769] [10] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv+0x1c2)[0x7f5126f1dbf5] [luna-0059:3835769] [11] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortz+0x1db)[0x7f5126f1ddea] [luna-0059:3835769] [12] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x408d5)[0x7f5126f028d5] [luna-0059:3835769] [13] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(mfu_flist_sort+0xa6)[0x7f5126f02b2f] [luna-0059:3835769] [14] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/mpifileutils/src/dwalk/dwalk(main+0xb8f)[0x55fe467c1e09] [luna-0059:3835769] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f5126a63083] [luna-0059:3835769] [16] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/mpifileutils/src/dwalk/dwalk(_start+0x2e)[0x55fe467c06ee] [luna-0059:3835769] *** End of error message *** [luna-0078:3879662:0:3879662] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x7f7465105000) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) ==== backtrace (tid:3879662) ==== 0 0x0000000000043090 killpg() ???:0 1 0x000000000018b9c9 __nss_database_lookup() ???:0 2 0x0000000000061012 dtcmp_merge_local_2way_memcpy() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_merge_2way.c:52 3 0x00000000000639f0 dtcmp_sort_local_mergesort_scratch() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:43 4 0x000000000006399c dtcmp_sort_local_mergesort_scratch() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:41 5 0x0000000000063b1f DTCMP_Sort_local_mergesort() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:77 6 0x000000000005b858 DTCMP_Sort_local() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1129 7 0x0000000000067828 DTCMP_Sortv_cheng_lwgrp() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sortv_cheng.c:759 8 0x00000000000679f9 DTCMP_Sortv_cheng() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sortv_cheng.c:809 9 0x000000000005bbf5 DTCMP_Sortv() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1291 10 0x000000000005bdea DTCMP_Sortz() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1365 11 0x00000000000408d5 sort_files_stat() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/common/mfu_flist_sort.c:545 12 0x0000000000040b2f mfu_flist_sort() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/common/mfu_flist_sort.c:612 13 0x0000000000003e09 main() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/dwalk/dwalk.c:670 14 0x0000000000024083 __libc_start_main() ???:0 15 0x00000000000026ee _start() ???:0 ================================= [luna-0078:3879662] *** Process received signal *** [luna-0078:3879662] Signal: Segmentation fault (11) [luna-0078:3879662] Signal code: (-6) [luna-0078:3879662] Failing at address: 0x3b32ee [luna-0078:3879662] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7f7497cd9090] [luna-0078:3879662] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x18b9c9)[0x7f7497e219c9] [luna-0078:3879662] [ 2] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(dtcmp_merge_local_2way_memcpy+0x158)[0x7f749817a012] [luna-0078:3879662] [ 3] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x639f0)[0x7f749817c9f0] [luna-0078:3879662] [ 4] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x6399c)[0x7f749817c99c] [luna-0078:3879662] [ 5] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sort_local_mergesort+0xf0)[0x7f749817cb1f] [luna-0078:3879662] [ 6] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sort_local+0xf8)[0x7f7498174858] [luna-0078:3879662] [ 7] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv_cheng_lwgrp+0x1a1)[0x7f7498180828] [luna-0078:3879662] [ 8] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv_cheng+0x7e)[0x7f74981809f9] [luna-0078:3879662] [ 9] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv+0x1c2)[0x7f7498174bf5] [luna-0078:3879662] [10] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortz+0x1db)[0x7f7498174dea] [luna-0078:3879662] [11] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x408d5)[0x7f74981598d5] [luna-0078:3879662] [12] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(mfu_flist_sort+0xa6)[0x7f7498159b2f] [luna-0078:3879662] [13] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/mpifileutils/src/dwalk/dwalk(main+0xb8f)[0x563af2fb2e09] [luna-0078:3879662] [14] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f7497cba083] [luna-0078:3879662] [15] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/mpifileutils/src/dwalk/dwalk(_start+0x2e)[0x563af2fb16ee] [luna-0078:3879662] *** End of error message *** [luna-0078:3879674:0:3879674] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x7f64701f4000) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0) ==== backtrace (tid:3879674) ==== 0 0x0000000000043090 killpg() ???:0 1 0x000000000018b9ce __nss_database_lookup() ???:0 2 0x0000000000060fe2 dtcmp_merge_local_2way_memcpy() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_merge_2way.c:49 3 0x00000000000639f0 dtcmp_sort_local_mergesort_scratch() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:43 4 0x000000000006399c dtcmp_sort_local_mergesort_scratch() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:41 5 0x00000000000639c1 dtcmp_sort_local_mergesort_scratch() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:42 6 0x0000000000063b1f DTCMP_Sort_local_mergesort() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sort_local_mergesort.c:77 7 0x000000000005b858 DTCMP_Sort_local() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1129 8 0x0000000000067828 DTCMP_Sortv_cheng_lwgrp() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sortv_cheng.c:759 9 0x00000000000679f9 DTCMP_Sortv_cheng() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp_sortv_cheng.c:809 10 0x000000000005bbf5 DTCMP_Sortv() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1291 11 0x000000000005bdea DTCMP_Sortz() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/dtcmp/src/dtcmp.c:1365 12 0x00000000000408d5 sort_files_stat() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/common/mfu_flist_sort.c:545 13 0x0000000000040b2f mfu_flist_sort() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/common/mfu_flist_sort.c:612 14 0x0000000000003e09 main() /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/mpifileutils/src/dwalk/dwalk.c:670 15 0x0000000000024083 __libc_start_main() ???:0 16 0x00000000000026ee _start() ???:0 ================================= [luna-0078:3879674] *** Process received signal *** [luna-0078:3879674] Signal: Segmentation fault (11) [luna-0078:3879674] Signal code: (-6) [luna-0078:3879674] Failing at address: 0x3b32fa [luna-0078:3879674] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7f649fa0b090] [luna-0078:3879674] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x18b9ce)[0x7f649fb539ce] [luna-0078:3879674] [ 2] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(dtcmp_merge_local_2way_memcpy+0x128)[0x7f649feabfe2] [luna-0078:3879674] [ 3] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x639f0)[0x7f649feae9f0] [luna-0078:3879674] [ 4] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x6399c)[0x7f649feae99c] [luna-0078:3879674] [ 5] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x639c1)[0x7f649feae9c1] [luna-0078:3879674] [ 6] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sort_local_mergesort+0xf0)[0x7f649feaeb1f] [luna-0078:3879674] [ 7] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sort_local+0xf8)[0x7f649fea6858] [luna-0078:3879674] [ 8] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv_cheng_lwgrp+0x1a1)[0x7f649feb2828] [luna-0078:3879674] [ 9] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv_cheng+0x7e)[0x7f649feb29f9] [luna-0078:3879674] [10] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortv+0x1c2)[0x7f649fea6bf5] [luna-0078:3879674] [11] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(DTCMP_Sortz+0x1db)[0x7f649fea6dea] [luna-0078:3879674] [12] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(+0x408d5)[0x7f649fe8b8d5] [luna-0078:3879674] [13] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/libmfu.so.4.0.0(mfu_flist_sort+0xa6)[0x7f649fe8bb2f] [luna-0078:3879674] [14] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/mpifileutils/src/dwalk/dwalk(main+0xb8f)[0x55691e9c1e09] [luna-0078:3879674] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f649f9ec083] [luna-0078:3879674] [16] /project/selene-admin/mpifileutils_debug/mpifileutils-v0.11.1/build/mpifileutils/src/dwalk/dwalk(_start+0x2e)[0x55691e9c06ee] [luna-0078:3879674] *** End of error message *** -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun noticed that process rank 30 with PID 0 on node luna-0059 exited on signal 11 (Segmentation fault). -------------------------------------------------------------------------- m