[WIP][test-java11] Test Hadoop 2.7 with JDK 11 #26533

org.apache.spark.sql.kafka010.KafkaSinkMicroBatchStreamingSuite.(It is not a test it is a sbt.testing.SuiteSelector)
org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite.(It is not a test it is a sbt.testing.SuiteSelector)

How to reproduce:

export JAVA_HOME=/usr/lib/jdk-11.0.3
export PATH=$JAVA_HOME/bin:$PATH

build/sbt "sql-kafka-0-10/test-only *.KafkaSinkMicroBatchStreamingSuite" -Phadoop-3.2 -Dhadoop.version=2.7.4 -Dcurator.version=2.7.1
build/sbt "sql-kafka-0-10/test-only *.KafkaDelegationTokenSuite" -Phadoop-3.2 -Dhadoop.version=2.7.4 -Dcurator.version=2.7.1

# Conflicts: # dev/deps/spark-deps-hadoop-3.2

SparkQA · 2019-11-15T07:13:44Z

Test build #113847 has finished for PR 26533 at commit 275390f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-11-15T07:29:28Z

@HyukjinKwon @BryanCutler It seems failed tests related to https://issues.apache.org/jira/browse/ARROW-5412?

…Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available

BryanCutler · 2019-11-15T07:55:25Z

right, I think for JDK >= 9 this needs to be set "-Dio.netty.tryReflectionSetAccessible=true". This was due to some Netty API that Arrow started using. I can look into it and try to figure out what we should do for Spark.

BryanCutler · 2019-11-15T07:57:01Z

Oh, I saw you just added it as a property in the pom. Hopefully that should do the trick.

SparkQA · 2019-11-15T08:05:02Z

Test build #113859 has finished for PR 26533 at commit a95e8bf.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

BryanCutler · 2019-11-15T08:54:36Z

retest this please

wangyum · 2019-11-15T09:23:32Z

@BryanCutler It seems not working:

export JAVA_HOME=/usr/lib/jdk-11.0.3
export PATH=$JAVA_HOME/bin:$PATH

build/sbt "sql/test-only *.ArrowConvertersSuite" -Phadoop-3.2 -Dio.netty.tryReflectionSetAccessible=true

SparkQA · 2019-11-15T10:42:33Z

Test build #113865 has finished for PR 26533 at commit a95e8bf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-11-16T06:13:40Z

@wangyum . Arrow broke all tests including PySpark/R. I'm working in the following PR. After fixing that, I'll ping you here.

[SPARK-29923][SQL][TESTS] Set io.netty.tryReflectionSetAccessible for Arrow on JDK9+ #26552

Please see the scala stuff there, too.

wangyum · 2019-11-16T11:35:22Z

Thank you @dongjoon-hyun

wangyum · 2019-11-16T11:35:49Z

retest this please

SparkQA · 2019-11-16T14:09:25Z

Test build #113926 has finished for PR 26533 at commit a95e8bf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-16T16:16:43Z

Test build #113935 has finished for PR 26533 at commit d0ffbd1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-16T16:39:02Z

Test build #113933 has finished for PR 26533 at commit 67c69db.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-11-18T23:57:45Z

All Failed Tests:

org.apache.spark.sql.kafka010.KafkaSinkMicroBatchStreamingSuite.(It is not a test it is a sbt.testing.SuiteSelector)
org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite.(It is not a test it is a sbt.testing.SuiteSelector)

How to reproduce:

export JAVA_HOME=/usr/lib/jdk-11.0.3
export PATH=$JAVA_HOME/bin:$PATH

build/sbt "sql-kafka-0-10/test-only *.KafkaSinkMicroBatchStreamingSuite" -Phadoop-3.2 -Dhadoop.version=2.7.4 -Dcurator.version=2.7.1
build/sbt "sql-kafka-0-10/test-only *.KafkaDelegationTokenSuite" -Phadoop-3.2 -Dhadoop.version=2.7.4 -Dcurator.version=2.7.1

The failed tests may be related to https://issues.apache.org/jira/browse/HADOOP-12911
cc @plusplusjiajia @AngersZhuuuu

SparkQA · 2019-11-19T08:05:01Z

Test build #114073 has finished for PR 26533 at commit eb50a4b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-11-19T08:05:20Z

retest this please

SparkQA · 2019-11-19T09:47:26Z

Test build #114077 has finished for PR 26533 at commit eb50a4b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-24T10:06:48Z

Test build #114338 has finished for PR 26533 at commit 50a11ec.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

# Conflicts: # pom.xml

SparkQA · 2019-11-24T11:30:55Z

Test build #114342 has finished for PR 26533 at commit 728952b.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-11-24T13:06:36Z

retest this please

SparkQA · 2019-11-24T13:46:34Z

Test build #114343 has finished for PR 26533 at commit 0a923c4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-24T15:46:42Z

Test build #114349 has finished for PR 26533 at commit 0a923c4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-28T17:10:08Z

Test build #114580 has finished for PR 26533 at commit be292f5.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2019-11-29T11:29:09Z

Test build #114624 has finished for PR 26533 at commit cbd22ed.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-12-05T06:20:01Z

The issue fixed by #26594

wangyum · 2020-08-17T09:37:03Z

Another issue:

[b_carmel@carmeldevour002-715563 spark-3.0.0-bin]$ bin/spark-sql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/b_carmel/yumwang/spark-3.1.0-SNAPSHOT-bin-benchmark-bloomfilter/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/apache/releases/hadoop-2.7.3.2.6.4-ebay.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/home/b_carmel/yumwang/spark-3.1.0-SNAPSHOT-bin-benchmark-bloomfilter/jars/spark-unsafe_2.12-3.1.0-SNAPSHOT.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "DataStreamer for file /tmp/hive-b_carmel/b_carmel/ff396890-3a46-4f2e-a93b-91de0eeefe0b/inuse.info" java.lang.NoSuchMethodError: 'sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner()'
	at org.apache.hadoop.crypto.CryptoStreamUtils.freeDB(CryptoStreamUtils.java:41)
	at org.apache.hadoop.crypto.CryptoInputStream.freeBuffers(CryptoInputStream.java:679)
	at org.apache.hadoop.crypto.CryptoInputStream.close(CryptoInputStream.java:315)
	at java.base/java.io.FilterInputStream.close(FilterInputStream.java:179)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeStream(DFSOutputStream.java:737)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeInternal(DFSOutputStream.java:685)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:680)
Exception in thread "DataStreamer for file /user/b_carmel/.sparkStaging/application_1595573410068_7264/__spark_libs__7269177261298078316.zip" java.lang.NoSuchMethodError: 'sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner()'
	at org.apache.hadoop.crypto.CryptoStreamUtils.freeDB(CryptoStreamUtils.java:41)
	at org.apache.hadoop.crypto.CryptoInputStream.freeBuffers(CryptoInputStream.java:679)
	at org.apache.hadoop.crypto.CryptoInputStream.close(CryptoInputStream.java:315)
	at java.base/java.io.FilterInputStream.close(FilterInputStream.java:179)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeStream(DFSOutputStream.java:737)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeInternal(DFSOutputStream.java:685)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:680)
Exception in thread "DataStreamer for file /user/b_carmel/.sparkStaging/application_1595573410068_7264/__spark_conf__.zip" java.lang.NoSuchMethodError: 'sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner()'
	at org.apache.hadoop.crypto.CryptoStreamUtils.freeDB(CryptoStreamUtils.java:41)
	at org.apache.hadoop.crypto.CryptoInputStream.freeBuffers(CryptoInputStream.java:679)
	at org.apache.hadoop.crypto.CryptoInputStream.close(CryptoInputStream.java:315)
	at java.base/java.io.FilterInputStream.close(FilterInputStream.java:179)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeStream(DFSOutputStream.java:737)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeInternal(DFSOutputStream.java:685)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:680)
Spark master: yarn, Application Id: application_1595573410068_7264

More details: https://issues.apache.org/jira/browse/HADOOP-12760

Test Hadoop 2.7 with JDK 11

e8e2cbc

Skip hadoop-cloud module:

2f0e6c8

https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-cloud-storage

Try to workaround: java.lang.IllegalArgumentException capacity < 0: (…

9811ad6

…-1 < 0)). https://stackoverflow.com/questions/58273063/pandasudf-and-pyarrow-0-15-0 http://arrow.apache.org/blog/2019/10/06/0.15.0-release/

wangyum added 2 commits November 15, 2019 13:02

Merge remote-tracking branch 'upstream/master' into hadoop-2.7-jdk11

600d093

# Conflicts: # dev/deps/spark-deps-hadoop-3.2

ignore KafkaSinkMicroBatchStreamingSuite and KafkaDelegationTokenSuite

275390f

Try to workaround: java.lang.UnsupportedOperationException: sun.misc.…

a95e8bf

…Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available

wangyum added 2 commits November 16, 2019 22:15

Remove workaround

67c69db

Revert ignored test

d0ffbd1

AngersZhuuuu mentioned this pull request Nov 19, 2019

[SPARK-29957][TEST] Reset MiniKDC's default enctypes to fit jdk8/jdk11 #26594

Closed

Upgrade minikdc version to 3.2.0

eb50a4b

HyukjinKwon mentioned this pull request Nov 21, 2019

[SPARK-29981][BUILD] Add hive-1.2/2.3 profiles #26619

Closed

Add refreshKrb5Config=true

50a11ec

Merge remote-tracking branch 'upstream/master' into hadoop-2.7-jdk11

728952b

# Conflicts: # pom.xml

Update

0a923c4

wangyum changed the title ~~[WIP][test-hadoop3.2][test-java11] Test Hadoop 2.7 with JDK 11~~ [WIP][test-java11] Test Hadoop 2.7 with JDK 11 Nov 24, 2019

fix

be292f5

wangyum added 2 commits November 29, 2019 08:48

Merge remote-tracking branch 'upstream/master' into hadoop-2.7-jdk11

54ad864

revert minikdc

cbd22ed

wangyum closed this Dec 5, 2019

wangyum deleted the hadoop-2.7-jdk11 branch August 17, 2020 09:37

[WIP][test-java11] Test Hadoop 2.7 with JDK 11 #26533

[WIP][test-java11] Test Hadoop 2.7 with JDK 11 #26533

Conversation

wangyum commented Nov 15, 2019

What changes were proposed in this pull request?

SparkQA commented Nov 15, 2019

SparkQA commented Nov 15, 2019

wangyum commented Nov 15, 2019

SparkQA commented Nov 15, 2019

SparkQA commented Nov 15, 2019

HyukjinKwon commented Nov 15, 2019

wangyum commented Nov 15, 2019

wangyum commented Nov 15, 2019

SparkQA commented Nov 15, 2019

wangyum commented Nov 15, 2019

BryanCutler commented Nov 15, 2019

BryanCutler commented Nov 15, 2019

SparkQA commented Nov 15, 2019

BryanCutler commented Nov 15, 2019

wangyum commented Nov 15, 2019

SparkQA commented Nov 15, 2019

dongjoon-hyun commented Nov 16, 2019 • edited Loading

wangyum commented Nov 16, 2019

wangyum commented Nov 16, 2019

SparkQA commented Nov 16, 2019

SparkQA commented Nov 16, 2019

SparkQA commented Nov 16, 2019

wangyum commented Nov 18, 2019

SparkQA commented Nov 19, 2019

wangyum commented Nov 19, 2019

SparkQA commented Nov 19, 2019

SparkQA commented Nov 24, 2019

SparkQA commented Nov 24, 2019

wangyum commented Nov 24, 2019

SparkQA commented Nov 24, 2019

SparkQA commented Nov 24, 2019

SparkQA commented Nov 28, 2019

SparkQA commented Nov 29, 2019

wangyum commented Dec 5, 2019

wangyum commented Aug 17, 2020

dongjoon-hyun commented Nov 16, 2019 •

edited

Loading