GH-40775: [Benchmarking][Java] Fix conbench timeout #40786

danepitkin · 2024-03-25T21:41:49Z

Rationale for this change

The java build script has been recently updated and it is affecting conbench, which is now seeing timeouts when building java. The logs are producing 100s of GB of data due to an unnecessary debug log msg.

What changes are included in this PR?

Delete log message on write to memory

Are these changes tested?

Yes, via conbench

Are there any user-facing changes?

No

GitHub Issue: [Benchmarking][Java] new java.lang.OutOfMemoryError in Java benchmarks after local build cache change #40775

github-actions · 2024-03-25T21:42:13Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

danepitkin · 2024-03-25T21:42:39Z

@ursabot please benchmark lang=Java

ursabot · 2024-03-25T21:42:44Z

Benchmark runs are scheduled for commit 17abf85. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.

conbench-apache-arrow · 2024-03-26T02:22:00Z

Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit 17abf85.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

vibhatha · 2024-03-26T06:49:09Z

dev/conbench_envs/benchmarks.env

@@ -48,3 +48,4 @@ PARQUET_BUILD_EXAMPLES=ON
 PARQUET_BUILD_EXECUTABLES=ON
 PYTHON=python
 LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
+JAVA_OPTIONS=-Xmx8G


@danepitkin just curious, should we set Xms as well?

Btw, would it be possible to make this CLI args? Just curious.

For Xms, I don't think it's necessary.

This can be passed in as cli args, too! We can add it as --java-options -Xmx8G in a different repo here: https://github.com/voltrondata-labs/benchmarks/blob/231078e715001c32d2f4ef2ffca27abb4625fd21/benchmarks/java_micro_benchmarks.py#L47-L58

For now, I'm using this method just to test more easily.

Thanks for trying this out, @danepitkin 🙂 If it turns out that tweaking the environment configuration fixes the error, I think leaving that configuration here (in the dev/conbench_envs location, which is only used by the Ursa machine) is the best choice.

Reason is we support running the voltrondata-labs/benchmarks in other locations too, like local development, so I wouldn't want to always clobber the memory options of users' machines with a CLI arg there.

Makes sense to me! Isolating the fix to the smallest scope is the preferred option IMO.

danepitkin · 2024-03-26T14:10:33Z

@ursabot please benchmark lang=Java

ursabot · 2024-03-26T14:10:37Z

Benchmark runs are scheduled for commit 6658872. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.

danepitkin · 2024-03-26T15:48:35Z

@ursabot please benchmark lang=Java

ursabot · 2024-03-26T15:48:39Z

Benchmark runs are scheduled for commit 97df26b. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.

danepitkin · 2024-03-26T15:55:44Z

Just noting here that setting java max heap size to 8GB throws


Exception in thread "CommonsExecStreamPumper-pool-9-thread-2" java.lang.OutOfMemoryError: Java heap space
--
  | at java.lang.StringCoding.decode(StringCoding.java:215)
  | at java.lang.String.<init>(String.java:463)
  | at java.lang.String.<init>(String.java:515)
  | at com.gradle.d.a.a.a(SourceFile:83)
  | at com.gradle.d.a.a.flush(SourceFile:78)
  | at com.gradle.d.a.a.write(SourceFile:72)
  | at java.io.PrintStream.write(PrintStream.java:480)
  | at com.gradle.d.a.b.write(SourceFile:203)
  | at org.apache.commons.exec.StreamPumper.run(StreamPumper.java:112)
  | at java.lang.Thread.run(Thread.java:750)

but increasing max heap size to 16GB throws


Exception in thread "CommonsExecStreamPumper-pool-9-thread-2" java.lang.OutOfMemoryError: GC overhead limit exceeded
--
  | at java.util.Arrays.copyOf(Arrays.java:3236)
  | at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:191)
  | at com.gradle.d.a.a.flush(SourceFile:78)
  | at com.gradle.d.a.a.write(SourceFile:72)
  | at java.io.PrintStream.write(PrintStream.java:480)
  | at com.gradle.d.a.b.write(SourceFile:203)
  | at org.apache.commons.exec.StreamPumper.run(StreamPumper.java:112)
  | at java.lang.Thread.run(Thread.java:750)

conbench-apache-arrow · 2024-03-26T16:11:57Z

Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit 6658872.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

danepitkin · 2024-03-26T17:58:56Z

@ursabot please benchmark lang=Java

ursabot · 2024-03-26T17:58:58Z

Benchmark runs are scheduled for commit 3d98b4a. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.

conbench-apache-arrow · 2024-03-26T18:12:30Z

Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit 97df26b.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

conbench-apache-arrow · 2024-03-27T00:32:49Z

Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit 3d98b4a.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

github-actions · 2024-03-27T19:28:04Z

⚠️ GitHub issue #40775 has been automatically assigned in GitHub to PR creator.

This reverts commit fce750c.

This reverts commit 63916d2.

This reverts commit 97df26b.

danepitkin · 2024-04-04T14:30:38Z

@ursabot please benchmark lang=Java

ursabot · 2024-04-04T14:30:41Z

Benchmark runs are scheduled for commit 7fafa78. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.

conbench-apache-arrow · 2024-04-04T17:04:01Z

Thanks for your patience. Conbench analyzed the 1 benchmarking run that has been run so far on PR commit 7fafa78.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

danepitkin · 2024-04-04T17:13:53Z

@ursabot please benchmark lang=Java

ursabot · 2024-04-04T17:13:58Z

Benchmark runs are scheduled for commit 7f2a480. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.

danepitkin · 2024-04-04T17:19:13Z

@ursabot please benchmark lang=Java

ursabot · 2024-04-04T17:19:18Z

Benchmark runs are scheduled for commit a09d83c. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.

conbench-apache-arrow · 2024-04-04T17:26:01Z

Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit 7f2a480.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

conbench-apache-arrow · 2024-04-04T18:13:44Z

Thanks for your patience. Conbench analyzed the 1 benchmarking run that has been run so far on PR commit a09d83c.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

danepitkin · 2024-04-04T18:18:51Z

java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestNettyAllocator.java

-          result);
+            result);
+      }
+


This was added to fix this bug that appeared in CI https://github.com/apache/arrow/actions/runs/8556425644/job/23446209816?pr=40786

danepitkin · 2024-04-04T18:19:54Z

@kou @vibhatha this is ready for final review!

lidavidm · 2024-04-04T22:17:14Z

@danepitkin mind updating the PR description now that the approach is different?

danepitkin · 2024-04-05T01:19:20Z

@danepitkin mind updating the PR description now that the approach is different?

Agh, good catch. Done!

kou

+1

conbench-apache-arrow · 2024-04-05T11:57:58Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 139afe5.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

### Rationale for this change The java build script has been recently updated and it is affecting conbench, which is now seeing timeouts when building java. The logs are producing 100s of GB of data due to an unnecessary debug log msg. ### What changes are included in this PR? * Delete log message on write to memory ### Are these changes tested? Yes, via conbench ### Are there any user-facing changes? No * GitHub Issue: apache#40775 Authored-by: Dane Pitkin <dane@voltrondata.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

WIP: Fix conbench timeout for java benchmarks

17abf85

danepitkin requested review from assignUser, kou and raulcd as code owners March 25, 2024 21:41

github-actions bot added the awaiting review Awaiting review label Mar 25, 2024

vibhatha reviewed Mar 26, 2024

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Mar 26, 2024

Bump max ram to 16GB

6658872

Enable HTTP connection pooling

97df26b

add maven clean to build command

3d98b4a

danepitkin changed the title ~~WIP: Fix conbench timeout for java benchmarks~~ GH-40775: Fix conbench timeout for java benchmarks Mar 27, 2024

Change log level to INFO

63916d2

danepitkin requested a review from lidavidm as a code owner March 28, 2024 00:07

github-actions bot added the Component: Java label Mar 28, 2024

danepitkin added 3 commits April 4, 2024 10:29

Revert "Put logback.xml in correct directory"

3e1ccb5

This reverts commit fce750c.

Revert "Change log level to INFO"

7dce0f9

This reverts commit 63916d2.

Revert "Enable HTTP connection pooling"

7fafa78

This reverts commit 97df26b.

danepitkin requested review from vibhatha and austin3dickey April 4, 2024 17:07

Fix ConcurrentModification bug in TestNettyAllocator.testMemoryUsage()

7f2a480

Fix indentation

a09d83c

danepitkin commented Apr 4, 2024

View reviewed changes

lidavidm approved these changes Apr 4, 2024

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Apr 4, 2024

kou approved these changes Apr 5, 2024

View reviewed changes

kou merged commit 139afe5 into apache:main Apr 5, 2024
19 checks passed

kou removed the awaiting merge Awaiting merge label Apr 5, 2024

kou mentioned this pull request Apr 5, 2024

[Benchmarking][Java] new java.lang.OutOfMemoryError in Java benchmarks after local build cache change #40775

Closed

GH-40775: [Benchmarking][Java] Fix conbench timeout #40786

GH-40775: [Benchmarking][Java] Fix conbench timeout #40786

Conversation

danepitkin commented Mar 25, 2024 • edited Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Mar 25, 2024

danepitkin commented Mar 25, 2024

ursabot commented Mar 25, 2024

conbench-apache-arrow bot commented Mar 26, 2024

vibhatha Mar 26, 2024

Choose a reason for hiding this comment

vibhatha Mar 26, 2024

Choose a reason for hiding this comment

danepitkin Mar 26, 2024

Choose a reason for hiding this comment

austin3dickey Mar 26, 2024

Choose a reason for hiding this comment

danepitkin Mar 26, 2024

Choose a reason for hiding this comment

danepitkin commented Mar 26, 2024

ursabot commented Mar 26, 2024

danepitkin commented Mar 26, 2024

ursabot commented Mar 26, 2024

danepitkin commented Mar 26, 2024

conbench-apache-arrow bot commented Mar 26, 2024

danepitkin commented Mar 26, 2024

ursabot commented Mar 26, 2024

conbench-apache-arrow bot commented Mar 26, 2024

conbench-apache-arrow bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

danepitkin commented Apr 4, 2024

ursabot commented Apr 4, 2024

conbench-apache-arrow bot commented Apr 4, 2024

danepitkin commented Apr 4, 2024

ursabot commented Apr 4, 2024

danepitkin commented Apr 4, 2024

ursabot commented Apr 4, 2024

conbench-apache-arrow bot commented Apr 4, 2024

conbench-apache-arrow bot commented Apr 4, 2024

danepitkin Apr 4, 2024

Choose a reason for hiding this comment

danepitkin commented Apr 4, 2024

lidavidm commented Apr 4, 2024

danepitkin commented Apr 5, 2024

kou left a comment

Choose a reason for hiding this comment

conbench-apache-arrow bot commented Apr 5, 2024

danepitkin commented Mar 25, 2024 •

edited

Loading