byteBuddyAgent installExternal cause jvm hangs

[heifrank]

Method of installExternal will create subProcess using ProcessBuilder.start(), and wait for subProcess to return.

SubProcess can of course write data, but when it writes relative more data, eg 64K, it can't write more because parent process does not read the data out. Since Parent process is waiting sub process to return, jvm got stuck.

Should ProcessBuilder in installExternal redirect stdout and stderr to null or start threads to consume the written data?

[raphw]

Normally, the attach should not write data back, but this indeed possible if data is written back. How do you encounter this problem?

The issue is that it would require a second thread to read the streams, given Java 5 APIs. But I will see how I can add this as an enhancement.

[heifrank]

Normally, the attach should not write data back, but this indeed possible if data is written back. How do you encounter this problem?

The issue is that it would require a second thread to read the streams, given Java 5 APIs. But I will see how I can add this as an enhancement.

@raphw java supports environment variable JAVA_TOOL_OPTIONS which serves the same functionality of -javaagent:xxx.jar . This is an convenient way for us because in some circumstances we cannot modify java startup command easily. Some java agents will write a bit more logs. BytebuddyAgent uses jvm attach(which is itself a java app), so the new java attach Process has also been influenced by JAVA_TOOL_OPTIONS.

[pepeshore]

@raphw when will this issue be closed, we encounter this problem too

[agibsonccc]

@raphw I'm seeing this too.
Edit: it looks like it was just a hidden stack trace.
I'll leave this for future users. Consider adding listeners
in situations where you have a hang.

Here's a sample:

Main" #1 prio=5 os_prio=0 cpu=587.37ms elapsed=107.47s tid=0x00007f27a647a000 nid=0x1edb79 runnable  [0x00007f27a8b1e000]
   java.lang.Thread.State: RUNNABLE
        at java.io.FileOutputStream.writeBytes(java.base@17.0.7/Native Method)
        at java.io.FileOutputStream.write(java.base@17.0.7/FileOutputStream.java:349)
        at java.io.BufferedOutputStream.flushBuffer(java.base@17.0.7/BufferedOutputStream.java:81)
        at java.io.BufferedOutputStream.flush(java.base@17.0.7/BufferedOutputStream.java:142)
        - locked <0x00000007776a88a8> (a java.io.BufferedOutputStream)
        at java.io.PrintStream.write(java.base@17.0.7/PrintStream.java:570)
        - locked <0x00000007776a8880> (a java.io.PrintStream)
        at sun.nio.cs.StreamEncoder.writeBytes(java.base@17.0.7/StreamEncoder.java:234)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(java.base@17.0.7/StreamEncoder.java:313)
        at sun.nio.cs.StreamEncoder.flushBuffer(java.base@17.0.7/StreamEncoder.java:111)
        - locked <0x00000007776a89f8> (a java.io.OutputStreamWriter)
        at java.io.OutputStreamWriter.flushBuffer(java.base@17.0.7/OutputStreamWriter.java:178)
        at java.io.PrintStream.write(java.base@17.0.7/PrintStream.java:699)
        - locked <0x00000007776a8880> (a java.io.PrintStream)
        at java.io.PrintStream.print(java.base@17.0.7/PrintStream.java:863)
        at java.io.PrintStream.append(java.base@17.0.7/PrintStream.java:1303)
        at java.io.PrintStream.append(java.base@17.0.7/PrintStream.java:63)
        at java.util.Formatter$FormatSpecifier.appendJustified(java.base@17.0.7/Formatter.java:3093)
        at java.util.Formatter$FormatSpecifier.print(java.base@17.0.7/Formatter.java:3083)
        at java.util.Formatter$FormatSpecifier.printString(java.base@17.0.7/Formatter.java:3056)
        at java.util.Formatter$FormatSpecifier.print(java.base@17.0.7/Formatter.java:2933)
        at java.util.Formatter.format(java.base@17.0.7/Formatter.java:2689)
        at java.io.PrintStream.format(java.base@17.0.7/PrintStream.java:1209)
        - locked <0x00000007776a8880> (a java.io.PrintStream)
        at java.io.PrintStream.printf(java.base@17.0.7/PrintStream.java:1105)
        at net.bytebuddy.agent.builder.AgentBuilder$Listener$StreamWriting.onDiscovery(AgentBuilder.java:1534)
        at net.bytebuddy.agent.builder.AgentBuilder$Listener$Compound.onDiscovery(AgentBuilder.java:1824)
        at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector.consider(AgentBuilder.java:7052)
        at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy.apply(AgentBuilder.java:4903)
        at net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(AgentBuilder.java:9574)
        at net.bytebuddy.agent.builder.AgentBuilder$Default.installOn(AgentBuilder.java:9495)
        at org.nd4j.interceptor.Nd4jInterceptor.premain(Nd4jInterceptor.java:84)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.7/Native Method)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.7/NativeMethodAccessorImpl.java:77)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.7/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@17.0.7/Method.java:568)
        at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(java.instrument@17.0.7/InstrumentationImpl.java:491)
        at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(java.instrument@17.0.7/InstrumentationImpl.java:503)

Of note could be the different JVM versions. I compiled this with java 11 but it's running in a java 17 JVM. I wonder if there's something going on there?

[raphw]

It should be compatible, but it's not guaranteed as far as I know.