PhaseAggressiveCoalesce SIGSEGV

[devinrsmith]

We've run into DB:test triggering SIGSEGV twice now in the same relative spot, on two different JDK 17 JVMs.

2021-10-30T06:24:33.1443208Z > Task :DB:test
2021-10-30T06:35:03.7824211Z #
2021-10-30T06:35:03.7924936Z # A fatal error has been detected by the Java Runtime Environment:
2021-10-30T06:35:03.8004601Z #
2021-10-30T06:35:03.8100735Z #  SIGSEGV (0xb) at pc=0x00007f641e67d27c, pid=9248, tid=9262
2021-10-30T06:35:03.8180524Z #
2021-10-30T06:35:03.8626465Z # JRE version: OpenJDK Runtime Environment Zulu17.28+13-CA (17.0+35) (build 17+35-LTS)
2021-10-30T06:35:03.8649483Z # Java VM: OpenJDK 64-Bit Server VM Zulu17.28+13-CA (17+35-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
2021-10-30T06:35:03.8659672Z # Problematic frame:
2021-10-30T06:35:03.8781830Z # V  [libjvm.so+0x5a327c]  PhaseAggressiveCoalesce::coalesce(Block*)+0x5bc
2021-10-30T06:35:03.8813477Z #
2021-10-30T06:35:03.8929788Z # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /home/runner/work/deephaven-core/core.9248)
2021-10-30T06:35:03.8949855Z #
2021-10-30T06:35:04.0609087Z # An error report file with more information is saved as:
2021-10-30T06:35:04.0635799Z # /home/runner/work/deephaven-core/hs_err_pid9248.log
2021-10-30T06:35:04.0664505Z #
2021-10-30T06:35:04.0712513Z # Compiler replay data is saved as:
2021-10-30T06:35:04.0759258Z # /home/runner/work/deephaven-core/replay_pid9248.log
2021-10-30T06:35:04.0821478Z #
2021-10-30T06:35:04.0898339Z # If you would like to submit a bug report, please visit:
2021-10-30T06:35:04.1025821Z #   http://www.azul.com/support/
2021-10-30T06:35:04.1072909Z #

and

2021-11-04T06:21:38.1020387Z > Task :DB:test
2021-11-04T06:32:54.1827737Z #
2021-11-04T06:32:54.1843776Z # A fatal error has been detected by the Java Runtime Environment:
2021-11-04T06:32:54.1851434Z #
2021-11-04T06:32:54.1862188Z #  SIGSEGV (0xb) at pc=0x00007f2dd7200dbc, pid=9554, tid=9568
2021-11-04T06:32:54.1869803Z #
2021-11-04T06:32:54.2121001Z # JRE version: OpenJDK Runtime Environment Temurin-17.0.1+12 (17.0.1+12) (build 17.0.1+12)
2021-11-04T06:32:54.2133835Z # Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.1+12 (17.0.1+12, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
2021-11-04T06:32:54.2142508Z # Problematic frame:
2021-11-04T06:32:54.2178345Z # V  [libjvm.so+0x5a2dbc]  PhaseAggressiveCoalesce::coalesce(Block*)+0x5bc
2021-11-04T06:32:54.2189617Z #
2021-11-04T06:32:54.2200460Z # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /home/runner/work/deephaven-core/core.9554)
2021-11-04T06:32:54.2208082Z #
2021-11-04T06:32:54.2220071Z # An error report file with more information is saved as:
2021-11-04T06:32:54.2229771Z # /home/runner/work/deephaven-core/hs_err_pid9554.log
2021-11-04T06:32:54.2239442Z #
2021-11-04T06:32:54.2247048Z # Compiler replay data is saved as:
2021-11-04T06:32:54.2258419Z # /home/runner/work/deephaven-core/replay_pid9554.log
2021-11-04T06:32:54.2268199Z #
2021-11-04T06:32:54.2276148Z # If you would like to submit a bug report, please visit:
2021-11-04T06:32:54.2291548Z #   https://github.com/adoptium/adoptium-support/issues
2021-11-04T06:32:54.2299594Z #

Unfortunately, we have been unable to grab the JVM err logs thus far - hopefully once #1528 we'll be able to capture the logs successfully.

[devinrsmith]

hs_err_pid9376.log
replay_pid9376.log

[devinrsmith]

Current CompileTask:
C2:  56727 16343       4       io.deephaven.db.v2.by.ssmpercentile.DoublePercentileTypeMedianHelper::setResult (228 bytes)

[devinrsmith]

hs_err_pid9228.log
replay_pid9228.log

Current CompileTask:
C2:  70711 17552 %     4       io.deephaven.db.v2.by.ssmpercentile.SsmChunkedPercentileOperator::addChunk @ 150 (328 bytes)

[devinrsmith]

Disabled JaCoCo to remove -javaagent; still happening:

hs_err_pid9512.log
replay_pid9512.log

hs_err_pid2501.log
replay_pid2501.log

[devinrsmith]

Another instance:

Current CompileTask:
C2: 379894 23743       4       io.deephaven.engine.table.impl.ssa.NullAwareCharSsaSsaStamp::processEntry (458 bytes)

Stack: [0x00007f41400a4000,0x00007f41401a5000],  sp=0x00007f41401a03f0,  free space=1008k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x6248bc]  PhaseAggressiveCoalesce::coalesce(Block*)+0x50c
V  [libjvm.so+0x624912]  PhaseCoalesce::coalesce_driver()+0x42
V  [libjvm.so+0x5ac6f0]  PhaseChaitin::Register_Allocate()+0x230
V  [libjvm.so+0x66b75a]  Compile::Code_Gen()+0x28a
V  [libjvm.so+0x66eefb]  Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool, bool, DirectiveSet*)+0xd4b
V  [libjvm.so+0x587184]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, DirectiveSet*)+0xd4
V  [libjvm.so+0x679364]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0x474
V  [libjvm.so+0x67ac68]  CompileBroker::compiler_thread_loop()+0x5a8
V  [libjvm.so+0xed38fa]  JavaThread::thread_main_inner()+0x1ba
V  [libjvm.so+0xed051f]  Thread::call_run()+0x14f
V  [libjvm.so+0xc7043e]  thread_native_entry(Thread*)+0xee

[devinrsmith]

Note: we've also seen this on the latest JDK 11 (qualitatively, less frequently, but hard to tell).

[niloc132]

Known failing methods:

io.deephaven.db.v2.by.ssmpercentile.DoublePercentileTypeMedianHelper::setResult
io.deephaven.db.v2.by.ssmpercentile.SsmChunkedPercentileOperator::addChunk
io.deephaven.db.v2.by.IncrementalChunkedByAggregationStateManager::buildTable
io.deephaven.db.v2.by.IncrementalChunkedOperatorAggregationStateManager::buildTable
io.deephaven.db.v2.RightIncrementalChunkedCrossJoinStateManager::addToIndex
io.deephaven.db.v2.IncrementalChunkedNaturalJoinStateManager::buildTable
io.deephaven.db.v2.IncrementalChunkedNaturalJoinStateManager::decorationProbe
io.deephaven.db.v2.IncrementalChunkedOperatorAggregationStateManager::buildTable

@rcaudy's research indicates that only C2 (aka tier 4) fails on these methods as far as we can tell. It is easier to reproduce this issue locally by passing -XX:Tier4CompileThreshold=1000 to the JVM. This jit compilation can be disabled by creating a compiler directives file like

[
    {
        match: ["io.deephaven.engine.table.impl.by.ssmpercentile.DoublePercentileTypeMedianHelper::setResult"],
        c2: {
            Exclude:true,
        },
    },
]

and passing it to java with a jvm arg like -XX:CompilerDirectivesFile=/path/to/c2-excludes.txt.

[rcaudy]

Note that the above packages are no longer correct.