ARROW-5412: [Integration] Add Java option for netty reflection

[BryanCutler]

After ARROW-3191, Java requires the property io.netty.tryReflectionSetAccessible to be set to true for JDK >= 9. This is already in the root POM, but causes integration tests to fail. This adds the property and an option to the Java command when running integration tests.

[BryanCutler]

@pravindra could you verify that this is correct for running integration tests with JDK 9 and above?
cc @emkornfield @bkietz

[praveenbingo]

@BryanCutler are the tests not part of CI? I am wondering why they did not fail with the ARROW-3191 changes.
Setting this property looks correct to me, looks like they are some indentation issues though.

[BryanCutler]

@BryanCutler are the tests not part of CI? I am wondering why they did not fail with the ARROW-3191 changes.

@praveenbingo CI uses JDK 8, which does not need this conf - I verified this locally too. When running integration tests locally with JDK 9 or higher, it will fail due to this.

[bkietz]

@BryanCutler This resolves the issue on my machine (oracle JDK11). Thanks!

[BryanCutler]

merged to master

[jnorthrup]

using oracle jdk12 i get

Exception in thread "main" java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available
at io.netty.util.internal.PlatformDependent.directBuffer(PlatformDependent.java:399)
at io.netty.buffer.NettyArrowBuf.getDirectBuffer(NettyArrowBuf.java:243)
at io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:233)
at io.netty.buffer.ArrowBuf.nioBuffer(ArrowBuf.java:245)
at org.apache.arrow.vector.ipc.message.ArrowRecordBatch.computeBodyLength(ArrowRecordBatch.java:218)
at org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:175)
at org.apache.arrow.vector.ipc.ArrowWriter.writeRecordBatch(ArrowWriter.java:119)
at org.apache.arrow.vector.ipc.ArrowFileWriter.writeRecordBatch(ArrowFileWriter.java:64)
at org.apache.arrow.vector.ipc.ArrowWriter.writeBatch(ArrowWriter.java:107)
at com.fnreport.QueryToFeather$Companion.go(QueryToFeather.kt:85)
at com.fnreport.QueryToFeather$Companion$main$1.invokeSuspend(QueryToFeather.kt:93)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:241)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:270)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:79)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:54)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:36)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at com.fnreport.QueryToFeather$Companion.main(QueryToFeather.kt:93)
at com.fnreport.QueryToFeather.main(QueryToFeather.kt)

my pom includes


org.glassfish.web
jstl-impl
1.2


com.google.code.gson
gson
2.8.5


postgresql
postgresql
9.1-901.jdbc4


mysql
mysql-connector-java
5.1.18


net.sourceforge.jtds
jtds
1.3.1


com.microsoft.sqlserver
mssql-jdbc
7.4.1.jre12

    <dependency>
        <groupId>org.xerial</groupId>
        <artifactId>sqlite-jdbc</artifactId>
        <version>3.28.0</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-core</artifactId>
        <version>[2.10,)</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>[2.10,)</version>
    </dependency>
    <dependency>
        <groupId>com.ibm.informix</groupId>
        <artifactId>jdbc</artifactId>
        <version>4.10.10</version>
    </dependency>
    <dependency>
        <groupId>org.jetbrains.kotlin</groupId>
        <artifactId>kotlin-stdlib-jdk8</artifactId>
        <version>${kotlin.version}</version>
    </dependency>
    <dependency>
        <groupId>org.jetbrains.kotlin</groupId>
        <artifactId>kotlin-test</artifactId>
        <version>${kotlin.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.jetbrains.kotlinx</groupId>
        <artifactId>kotlinx-coroutines-core</artifactId>
        <version>1.3.0-RC</version>
    </dependency><!-- https://mvnrepository.com/artifact/org.apache.arrow/arrow-jdbc -->

org.apache.arrow arrow-jdbc 0.14.1

[emkornfield]

@jnorthrup I created https://issues.apache.org/jira/browse/ARROW-6206 to track documenting this for consumers of the Java library. I don't think the library should be automatically setting the property, but if you have feeling on this, feel free to discuss on the Arrow Dev mailing list.

[jnorthrup]

@emkornfield im unclear on the priority to use NIO directbytebuffers at this time, if the goal is not to explicitly document and take advantage of memory map files, and or provide an optional off-heap option. for writing records to disk serially, NIO is killing a fly with a sledge-hammer and opens up the package to enormous bug surface area in the unmentionables of misc.unsafe.

since I am exclusively interacting with ways to get jdbc to a file format that python can use I should be unencumbered by NIO/netty instabilities.

wrt setting properties, i have no opinion. i set the parameter in my jdbc client in the hopes that it fixes a temporary speedbump and the property will not matter if and when the problem goes away..