Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

openjdk11_j9_extended.functional_ppc64le_linux_xl testOpenJ9DiagnosticsMXBean_1 Segmentation error vmState=0x00020002 #8019

Closed
JasonFengJ9 opened this issue Dec 8, 2019 · 13 comments
Labels
comp:vm segfault Issues that describe segfaults / JVM crashes test failure

Comments

@JasonFengJ9
Copy link
Member

Failure link

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_extended.functional_ppc64le_linux_xl_Nightly/8/tapResults/

Optional info

Failure output (captured from console output)

Unhandled exception
Type=Segmentation error vmState=0x00020002
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00003FFF7CBA64D0 Handler2=00003FFF7C91F2A0
R0=0000000000000010 R1=00003FFEBF62BE60 R2=00003FFF7C3E5700 R3=00003FFEF7B95928
R4=0000000000000009 R5=0000000000000003 R6=0000000000000000 R7=00003FFEF5426D48
R8=00003FFEF5426D50 R9=00003FFEF7FD7C90 R10=0000000000002000 R11=FFFFFFFFFFFFFFFF
R12=00003FFF7C152BC0 R13=00003FFEBF636900 R14=FFFFFFFFFFFFFFFF R15=0000000099669966
R16=00003FFEF7FD7C90 R17=0000000000000000 R18=00003FFF780487A0 R19=0000000000000010
R20=0000000000000090 R21=00000000000004B8 R22=00003FFE88004180 R23=00003FFEBF62BEF8
R24=00003FFF7C395608 R25=0000000000000000 R26=0000000000000001 R27=00003FFEBF62BED0
R28=00003FFEBF62BEE8 R29=00003FFEBF62BEE0 R30=00003FFEBF62BEC8 R31=00003FFE88003FA8
NIP=00003FFF7C23500C MSR=800000000280F033 ORIG_GPR3=00003FFF7C28C8F4 CTR=00003FFF7C152BC0
LINK=00003FFF7C235508 XER=0000000000000000 CCR=0000000044224844 SOFTE=0000000000000001
TRAP=0000000000000300 DAR=0000000000000000 dsisr=0000000040000000 RESULT=0000000000000000
FPR0 0000000000000001 (f: 1.000000, d: 4.940656e-324)
FPR1 4027f367bcbfbe41 (f: 3166682624.000000, d: 1.197540e+01)
FPR2 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
FPR3 3c56000000000000 (f: 0.000000, d: 4.770490e-18)
FPR4 402791272ee9db80 (f: 787078016.000000, d: 1.178350e+01)
FPR5 3ff3627f60000000 (f: 1610612736.000000, d: 1.211547e+00)
FPR6 4103628000000000 (f: 0.000000, d: 1.588000e+05)
FPR7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR8 3cbc000000000000 (f: 0.000000, d: 3.885781e-16)
FPR9 3d7072fdfc9e494f (f: 4238231808.000000, d: 9.350281e-13)
FPR10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR13 3ea4000000000000 (f: 0.000000, d: 5.960464e-07)
FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/home/jenkins/jenkins-agent/workspace/Test_openjdk11_j9_extended.functional_ppc64le_linux_xl_Nightly/openjdkbinary/j2sdk-image/lib/default/libj9gc29.so
Module_base_address=00003FFF7C0C0000
Target=2_90_20191208_10 (Linux 4.4.0-145-generic)
CPU=ppc64le (4 logical CPUs) (0x1fe3b0000 RAM)
----------- Stack Backtrace -----------
(0x00003FFF7C23500C [libj9gc29.so+0x17500c])
(0x00003FFF7C235508 [libj9gc29.so+0x175508])
(0x00003FFF7C235F68 [libj9gc29.so+0x175f68])
(0x00003FFF7C235FB8 [libj9gc29.so+0x175fb8])
(0x00003FFF7C3458B0 [libj9gc29.so+0x2858b0])
(0x00003FFF7C246528 [libj9gc29.so+0x186528])
(0x00003FFF7C246008 [libj9gc29.so+0x186008])
(0x00003FFF7C9206A8 [libj9prt29.so+0x306a8])
(0x00003FFF7C245890 [libj9gc29.so+0x185890])
(0x00003FFF7CAD1148 [libj9thr29.so+0x11148])
(0x00003FFF7D5B8040 [libpthread.so.0+0x8040])
clone+0x98 (0x00003FFF7D783BB0 [libc.so.6+0x123bb0])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2019/12/08 11:02:57 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/jenkins-agent/workspace/Test_openjdk11_j9_extended.functional_ppc64le_linux_xl_Nightly/openjdk-tests/TKG/test_output_15757950549105/testOpenJ9DiagnosticsMXBean_1/core.20191208.110257.22328.0001.dmp' in response to an event
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.22430.

JVMDUMP010I System dump written to /home/jenkins/jenkins-agent/workspace/Test_openjdk11_j9_extended.functional_ppc64le_linux_xl_Nightly/openjdk-tests/TKG/test_output_15757950549105/testOpenJ9DiagnosticsMXBean_1/core.20191208.110257.22328.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/jenkins/jenkins-agent/workspace/Test_openjdk11_j9_extended.functional_ppc64le_linux_xl_Nightly/openjdk-tests/TKG/test_output_15757950549105/testOpenJ9DiagnosticsMXBean_1/javacore.20191208.110257.22328.0002.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/jenkins-agent/workspace/Test_openjdk11_j9_extended.functional_ppc64le_linux_xl_Nightly/openjdk-tests/TKG/test_output_15757950549105/testOpenJ9DiagnosticsMXBean_1/javacore.20191208.110257.22328.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/jenkins-agent/workspace/Test_openjdk11_j9_extended.functional_ppc64le_linux_xl_Nightly/openjdk-tests/TKG/test_output_15757950549105/testOpenJ9DiagnosticsMXBean_1/Snap.20191208.110257.22328.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/jenkins-agent/workspace/Test_openjdk11_j9_extended.functional_ppc64le_linux_xl_Nightly/openjdk-tests/TKG/test_output_15757950549105/testOpenJ9DiagnosticsMXBean_1/Snap.20191208.110257.22328.0003.trc
JVMDUMP007I JVM Requesting JIT dump using '/home/jenkins/jenkins-agent/workspace/Test_openjdk11_j9_extended.functional_ppc64le_linux_xl_Nightly/openjdk-tests/TKG/test_output_15757950549105/testOpenJ9DiagnosticsMXBean_1/jitdump.20191208.110257.22328.0004.dmp'
JVMDUMP010I JIT dump written to /home/jenkins/jenkins-agent/workspace/Test_openjdk11_j9_extended.functional_ppc64le_linux_xl_Nightly/openjdk-tests/TKG/test_output_15757950549105/testOpenJ9DiagnosticsMXBean_1/jitdump.20191208.110257.22328.0004.dmp
JVMDUMP013I Processed dump event "gpf", detail "".

testOpenJ9DiagnosticsMXBean_1_FAILED
@pshipton
Copy link
Member

pshipton commented Dec 9, 2019

@dmitripivkine can you pls take a look

@dmitripivkine
Copy link
Contributor

This crash occur because of Ljava/lang/Module; module = !fj9object 0x3ffef7fd7c90 in class object !j9object 0x00003FFEF7B95868 class !j9class 0x00003FFF787AEC00 (jdk/jmx/remote/internal/rmi/PRef) has been collected

@dmitripivkine
Copy link
Contributor

I have result files downloaded, ask me where

@dmitripivkine
Copy link
Contributor

In order to discover module objects GC walks classLoader->moduleHashTable !j9classloader 0x00003FFF7808EDA8 moduleHashTable = !j9hashtable 0x00003FFF78533740

@dmitripivkine
Copy link
Contributor

dmitripivkine commented Dec 10, 2019

Looks like at the moment of crash module address is added to the hash table. So I guess module was collected right after creation but before added to hash table. We should double check there is no time when GC might occur

@pshipton
Copy link
Member

This is a low reproduction rate (only reported once so far), and it's getting to be to late to make any more fixes in the 0.18 release, moving it out.

@pshipton
Copy link
Member

Seems this won't be resolved for the 0.20 release, moving it forward.

@M-Davies
Copy link

Seen at Adopt on https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_sanity.openjdk_aarch64_linux/103/#showFailuresLink
java/util/stream/boottest/java.base/java/util/stream/SliceSpliteratorTest.java

19:27:18  [2020-03-30 18:27:17,825] Agent[24]: stderr: 000000000AA2EA00: J9_INVALID_OBJECT in thread pool-1-thread-1
19:27:18  [2020-03-30 18:27:17,825] Agent[24]: stderr: 000000000AA2EA00:	O-Slot=0000FFFF75578008
19:27:18  [2020-03-30 18:27:17,825] Agent[24]: stderr: 000000000AA2EA00:	O-Slot value=FFFFFFFFFFFFFFFF
19:27:18  [2020-03-30 18:27:17,825] Agent[24]: stderr: 000000000AA2EA00:	PC=0000FFFF781A182C
19:27:18  [2020-03-30 18:27:17,826] Agent[24]: stderr: 000000000AA2EA00:	framesWalked=3
19:27:18  [2020-03-30 18:27:17,826] Agent[24]: stderr: 000000000AA2EA00:	arg0EA=000000000AAF9DB0
19:27:18  [2020-03-30 18:27:17,826] Agent[24]: stderr: 000000000AA2EA00:	walkSP=000000000AAF9CA8
19:27:18  [2020-03-30 18:27:17,826] Agent[24]: stderr: 000000000AA2EA00:	literals=000000000AAE4350
19:27:18  [2020-03-30 18:27:17,826] Agent[24]: stderr: 000000000AA2EA00:	jitInfo=0000FFFF772F7538
19:27:18  [2020-03-30 18:27:17,826] Agent[24]: stderr: Unhandled exception
19:27:18  [2020-03-30 18:27:17,826] Agent[24]: stderr: Type=Segmentation error vmState=0x00020002
19:27:18  [2020-03-30 18:27:17,826] Agent[24]: stderr: J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
19:27:18  [2020-03-30 18:27:17,826] Agent[24]: stderr: Handler1=0000FFFF95350014 Handler2=0000FFFF9522DB28 InaccessibleAddress=0000FFFE38033418
19:27:18  [2020-03-30 18:27:17,827] Agent[24]: stderr: R0=0000FFFF8FFCFF68 R1=0000FFFE3803341A R2=0000FFFE38033418 R3=000000000AE2BD10
19:27:18  [2020-03-30 18:27:17,827] Agent[24]: stderr: R4=0000000000009A81 R5=0000FFFE1C0197FA R6=0000000000000000 R7=0000FFFE1C019A8E
19:27:18  [2020-03-30 18:27:17,827] Agent[24]: stderr: R8=00000000000000A9 R9=0000000000000002 R10=0000FFFF75577A70 R11=0000000000000001
19:27:18  [2020-03-30 18:27:17,827] Agent[24]: stderr: R12=0000FFFF755778F1 R13=FFFFFFFFFFFFFFFF R14=0000000000000005 R15=0000000000000030
19:27:18  [2020-03-30 18:27:17,827] Agent[24]: stderr: R16=0000FFFF94ACD000 R17=0000FFFF95E12190 R18=0000000000000002 R19=0000FFFF954A0E18
19:27:18  [2020-03-30 18:27:17,827] Agent[24]: stderr: R20=000000000AA2EA00 R21=0000FFFF75577F50 R22=0000FFFF20001CC8 R23=0000FFFE1C019A8C
19:27:18  [2020-03-30 18:27:17,827] Agent[24]: stderr: R24=000000001C019988 R25=0000FFFE1C019A90 R26=0000000000000040 R27=0000FFFE1C0197B8
19:27:18  [2020-03-30 18:27:17,828] Agent[24]: stderr: R28=0000000000000000 R29=0000FFFF75577E80 R30=0000FFFF8FF54344 R31=0000FFFF75577E60
19:27:18  [2020-03-30 18:27:17,828] Agent[24]: stderr: PC=0000FFFF8FF543D8 SP=0000FFFF75577E60 PSTATE=0000000020000000
19:27:18  [2020-03-30 18:27:17,828] Agent[24]: stderr: V0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,828] Agent[24]: stderr: V1 0000000000000001 (f: 1.000000, d: 4.940656e-324)
19:27:18  [2020-03-30 18:27:17,828] Agent[24]: stderr: V2 ffffffff00000000 (f: 0.000000, d: -nan)
19:27:18  [2020-03-30 18:27:17,828] Agent[24]: stderr: V3 0000000000000004 (f: 4.000000, d: 1.976263e-323)
19:27:18  [2020-03-30 18:27:17,828] Agent[24]: stderr: V4 ffffffffffffffff (f: 4294967296.000000, d: -nan)
19:27:18  [2020-03-30 18:27:17,828] Agent[24]: stderr: V5 0000000000000001 (f: 1.000000, d: 4.940656e-324)
19:27:18  [2020-03-30 18:27:17,829] Agent[24]: stderr: V6 0000000100000001 (f: 1.000000, d: 2.121996e-314)
19:27:18  [2020-03-30 18:27:17,829] Agent[24]: stderr: V7 ffffffffffffffff (f: 4294967296.000000, d: -nan)
19:27:18  [2020-03-30 18:27:17,829] Agent[24]: stderr: V8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,829] Agent[24]: stderr: V9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,829] Agent[24]: stderr: V10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,829] Agent[24]: stderr: V11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,829] Agent[24]: stderr: V12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,829] Agent[24]: stderr: V13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,830] Agent[24]: stderr: V14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,830] Agent[24]: stderr: V15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,830] Agent[24]: stderr: V16 fffffffcfffffffc (f: 4294967296.000000, d: -nan)
19:27:18  [2020-03-30 18:27:17,830] Agent[24]: stderr: V17 c037154748bef6c8 (f: 1220474624.000000, d: -2.308312e+01)
19:27:18  [2020-03-30 18:27:17,830] Agent[24]: stderr: V18 401e840b4ac4e4d2 (f: 1254417664.000000, d: 7.628949e+00)
19:27:18  [2020-03-30 18:27:17,830] Agent[24]: stderr: V19 0000ffff942a67f0 (f: 2485807104.000000, d: 1.390662e-309)
19:27:18  [2020-03-30 18:27:17,830] Agent[24]: stderr: V20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,830] Agent[24]: stderr: V21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,830] Agent[24]: stderr: V22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,831] Agent[24]: stderr: V23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,831] Agent[24]: stderr: V24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,831] Agent[24]: stderr: V25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,831] Agent[24]: stderr: V26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,831] Agent[24]: stderr: V27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,831] Agent[24]: stderr: V28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,831] Agent[24]: stderr: V29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,831] Agent[24]: stderr: V30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,832] Agent[24]: stderr: V31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
19:27:18  [2020-03-30 18:27:17,832] Agent[24]: stderr: Module=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_linux/openjdkbinary/j2sdk-image/jdk-11.0.7+9/lib/compressedrefs/libj9gc29.so
19:27:18  [2020-03-30 18:27:17,832] Agent[24]: stderr: Module_base_address=0000FFFF8FD9C000
19:27:18  [2020-03-30 18:27:17,832] Agent[24]: stderr: Target=2_90_20200330_255 (Linux 4.15.0-34-generic)
19:27:18  [2020-03-30 18:27:17,832] Agent[24]: stderr: CPU=aarch64 (64 logical CPUs) (0x1f699a1000 RAM)
19:27:18  [2020-03-30 18:27:17,832] Agent[24]: stderr: ----------- Stack Backtrace -----------
19:27:18  [2020-03-30 18:27:17,832] Agent[24]: stderr: (0x0000FFFF9524B530 [libj9prt29.so+0x37530])
19:27:18  [2020-03-30 18:27:17,832] Agent[24]: stderr: ---------------------------------------

Dumps: https://ibm.box.com/shared/static/xli15fximvjrc72tf7c4s4fmhhmd5wqh.gz

@DanHeidinga
Copy link
Member

@M-Davies @JasonFengJ9 Does this issue still occur?

@JasonFengJ9
Copy link
Member Author

@DanHeidinga No, I don't recall such failure recently, also searched last ten Pipeline-Build-Test-All-11 builds in https://ci.eclipse.org/openj9 via TRSS and didn't have a hit.
So it can be closed for working on later builds unless @M-Davies disagrees.

@M-Davies
Copy link

I'm running some grinders to verify but I don't think so unless you've seen it recently @archanaNogriya ?

@DanHeidinga
Copy link
Member

I'm going to close this now given the lack of reoccurrence. If it happens again, please comment here and the issue can be reopened

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:vm segfault Issues that describe segfaults / JVM crashes test failure
Projects
None yet
Development

No branches or pull requests

6 participants