EPIC: PhaseAggressiveCoalesce::coalesce triage

[devinrsmith]

Due to #1529 we've added a compiler directives file; we need to work on continued "triage" of this issue, and trying to produce a minimal reproducer so we can properly file an OpenJDK bug report.

I'll collect unique versions and hosts where this has been seen (as reported by the hs_err_pid files attached).

Versions (demonstrated against latest versions of OpenJDK 11, 17, and 18):

# JRE version: OpenJDK Runtime Environment Temurin-18.0.1+10 (18.0.1+10) (build 18.0.1+10)
# Java VM: OpenJDK 64-Bit Server VM Temurin-18.0.1+10 (18.0.1+10, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x59e8f5]  PhaseAggressiveCoalesce::coalesce(Block*)+0x65

# JRE version: OpenJDK Runtime Environment Temurin-11.0.15+10 (11.0.15+10) (build 11.0.15+10)
# Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.15+10 (11.0.15+10, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x62619c]  PhaseAggressiveCoalesce::coalesce(Block*)+0x50c

# JRE version: OpenJDK Runtime Environment Temurin-17.0.3+7 (17.0.3+7) (build 17.0.3+7)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.3+7 (17.0.3+7, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x597885]  PhaseAggressiveCoalesce::coalesce(Block*)+0x65

# JRE version: OpenJDK Runtime Environment Temurin-17.0.1+12 (17.0.1+12) (build 17.0.1+12)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.1+12 (17.0.1+12, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x5a2dbc]  PhaseAggressiveCoalesce::coalesce(Block*)+0x5bc

# JRE version: OpenJDK Runtime Environment Zulu11.54+23-CA (11.0.14+9) (build 11.0.14+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu11.54+23-CA (11.0.14+9-LTS, mixed mode, tiered, compressed oops, g1 gc, bsd-aarch64)
# Problematic frame:
# V  [libjvm.dylib+0x1923dc]  _ZN23PhaseAggressiveCoalesce8coalesceEP5Block+0x58

# JRE version: OpenJDK Runtime Environment Zulu11.52+13-CA (11.0.13+8) (build 11.0.13+8-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu11.52+13-CA (11.0.13+8-LTS, mixed mode, tiered, compressed oops, g1 gc, bsd-aarch64)
# Problematic frame:
# V  [libjvm.dylib+0x1931ac]  _ZN23PhaseAggressiveCoalesce8coalesceEP5Block+0x58

CI hosts:

• Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz, 2 cores, 6G, Ubuntu 20.04.4 LTS
• Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz, 2 cores, 6G, Ubuntu 20.04.4 LTS
• Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz, 2 cores, 6G, Ubuntu 20.04.4 LTS
• Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz, 2 cores, 6G, Ubuntu 20.04.4 LTS
• Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz, 2 cores, 6G, Ubuntu 20.04.3 LTS
• Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz, 2 cores, 6G, Ubuntu 20.04.3 LTS
• Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz, 2 cores, 6G, Ubuntu 20.04.3 LTS

Developer hosts:

• AMD Ryzen 9 5900X 12-Core Processor, 24 cores, 125G, Fedora release 35 (Thirty Five)
• MacBookPro18,2 arm64 1 MHz, 10 cores, 64G, Darwin 21.1.0
• MacBookPro18,2 arm64 1 MHz, 10 cores, 64G, Darwin 21.3.0

Note: some of the error files show jacoco added as a javaagent. This was subsequently disabled to rule out jacoco as a cause.

Reproduce locally:

Branch https://github.com/deephaven/deephaven-core/tree/nightly/phase-aggressive-sigsegv

./gradlew -PtestRuntimeVersion=17 -PtestRuntimeVendor=adoptopenjdk -PforceTest=true engine-table:testOutOfBand --tests io.deephaven.engine.table.impl.QueryTableAggregationTest.testMedianByIncremental

[devinrsmith]

Here are some of the error logs. There are many that haven't been uploaded. I will try to collect more here as the happen.

hs_err_pid9376.log
hs_err_pid9228.log
hs_err_pid9512.log
hs_err_pid2501.log
hs_err_pid8315.log
hs_err_pid8553.log
hs_err_pid8895.log

[devinrsmith]

hs_err_pid4360.log

[devinrsmith]

hs_err_pid3267.log

[devinrsmith]

My personal box: hs_err_pid155691.log

[devinrsmith]

Demo against Temurin-18.0.1+10: hs_err_pid174576.log

[devinrsmith]

https://bugs.openjdk.org/browse/JDK-8287432