GH-43377: [Java][CI] Java-Jars CI is Failing with a linking error on macOS #43385

vibhatha · 2024-07-23T08:42:17Z

Rationale for this change

For googletest, we have installation via BUNDLED and brew, a version mismatch from one of these options could cause conflicts and linking related issues. Preferring BUNDLED version, this PR uninstalls the brew installation of googletest.

What changes are included in this PR?

Removing brew installation of googletest

Are these changes tested?

Yes.

Are there any user-facing changes?

No

GitHub Issue: [Java][CI] Java-Jars CI is Failing with a linking error on macOS #43377

vibhatha · 2024-07-23T08:42:39Z

@github-actions crossbow submit -g java

github-actions · 2024-07-23T08:42:45Z

⚠️ GitHub issue #43377 has been automatically assigned in GitHub to PR creator.

github-actions · 2024-07-23T08:45:22Z

Revision: 02eb021

Submitted crossbow builds: ursacomputing/crossbow @ actions-5dda3b0a91

Task	Status
java-jars
test-conda-python-3.11-spark-master
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

llama90

I reviewed it quickly. Thank you for looking into the issue caused by my PR.

dev/tasks/java-jars/github.yml

llama90 · 2024-07-23T09:02:15Z

Additionally, according to the link, it seems that the MACOSX_DEPLOYMENT_TARGET needs to be updated, but I’ve noticed that this code appears in many places.

I think this should be addressed beyond just Java. Should I create an issue for this?

code search result

vibhatha · 2024-07-23T09:03:39Z

Additionally, according to the link, it seems that the MACOSX_DEPLOYMENT_TARGET needs to be updated, but I’ve noticed that this code appears in many places.

I think this should be addressed beyond just Java. Should I create an issue for this?

That would be wonderful @llama90

* [code search result](https://github.com/search?q=repo%3Aapache%2Farrow%20MACOSX_DEPLOYMENT_TARGET&type=code)

llama90 · 2024-07-23T09:48:54Z

FYI: #43386 (comment)

vibhatha · 2024-07-23T10:28:53Z

@github-actions crossbow submit -g java java-jars

github-actions · 2024-07-23T10:31:10Z

Revision: 4b467bd

Submitted crossbow builds: ursacomputing/crossbow @ actions-9387c4cfe5

Task	Status
java-jars
test-conda-python-3.11-spark-master
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

vibhatha · 2024-07-23T10:36:40Z

@github-actions crossbow submit java-jars

github-actions · 2024-07-23T10:38:59Z

Revision: 4b467bd

Submitted crossbow builds: ursacomputing/crossbow @ actions-834fce646a

Task	Status
java-jars

danepitkin · 2024-07-23T23:25:40Z

I don't think changing MACOSX_DEPLOYMENT_TARGET is going to help here, only because the warning also appears in the last successful java-jars build. It looks like parquet tests and orc tests are failing to link in the C++ layer.

For example, in https://github.com/ursacomputing/crossbow/actions/runs/10055540527/job/27792410574:

...
[497/510] Linking CXX executable release/arrow-orc-adapter-test
FAILED: release/arrow-orc-adapter-test 
...
[498/510] Linking CXX executable release/parquet-internals-test
FAILED: release/parquet-internals-test 
...
[499/510] Linking CXX executable release/parquet-writer-test
FAILED: release/parquet-writer-test 
...
[500/510] Linking CXX executable release/parquet-reader-test
FAILED: release/parquet-reader-test 
...
[501/510] Linking CXX executable release/parquet-arrow-test
FAILED: release/parquet-arrow-test 
...

@kou by chance do you know what might be happening here?

vibhatha · 2024-07-24T00:03:23Z

@danepitkin I was merely testing an idea, in the time period the change happened the direct change I could find was the change in the CIs. And I agree, the issue is with the test library which cannot be linked.

kou · 2024-07-24T01:11:28Z

It seems that this was happen since 2024-07-16: https://github.com/ursacomputing/crossbow/actions/runs/9953458018/job/27502447467
This was not happen at 2024-07-15: https://github.com/ursacomputing/crossbow/actions/runs/9936214033/job/27449403626

We may be able to find a cause of this by diffing these logs.

vibhatha · 2024-07-25T08:46:16Z

@kou I looked into the history of merged PRs and sort of tracked from where it started occurring. It seems like we haven't made significant changes to the build or C++ components from Java PRs.

At this point #43109 (comment) the java-jars failed at merge for a different reason not an issue with MacOS related builds. But when I re-ran the crossbow on the merged PR, I get the current issue we are experiencing. Also this failing CI is only related to Java as it seems, and existing C++ CIs have not been reported failing AFAIK. Could this be an environment related issue?

vibhatha · 2024-07-25T09:19:30Z

@github-actions crossbow submit -g cpp

vibhatha · 2024-07-25T09:20:51Z

Just saw this filed #43400, this is the same issue.

github-actions · 2024-07-25T09:21:57Z

Revision: 4b467bd

Submitted crossbow builds: ursacomputing/crossbow @ actions-214fbb71e3

Task	Status
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-14

vibhatha · 2024-07-25T09:35:17Z

@github-actions crossbow submit -g java

github-actions · 2024-07-25T09:37:38Z

Revision: 5879052

Submitted crossbow builds: ursacomputing/crossbow @ actions-0883833aa3

Task	Status
java-jars
test-conda-python-3.11-spark-master
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

vibhatha · 2024-07-25T10:05:22Z

@github-actions crossbow submit java-jars

github-actions · 2024-07-25T10:07:44Z

Revision: 51f0764

Submitted crossbow builds: ursacomputing/crossbow @ actions-04d5e09e65

Task	Status
java-jars

vibhatha · 2024-07-25T23:29:41Z

@github-actions crossbow submit java-jars

github-actions · 2024-07-25T23:32:13Z

Revision: 85c1244

Submitted crossbow builds: ursacomputing/crossbow @ actions-6f3ab99e36

Task	Status
java-jars

vibhatha · 2024-07-26T00:01:33Z

@github-actions crossbow submit -g java

github-actions · 2024-07-26T00:03:47Z

Revision: 10c59a8

Submitted crossbow builds: ursacomputing/crossbow @ actions-96d6d76c00

Task	Status
java-jars
test-conda-python-3.11-spark-master
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

danepitkin

LGTM, amazing work! I have always used bundled GTest when developing on MacOS, so I think this is good to include in our CI.

conbench-apache-arrow · 2024-07-26T23:37:37Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 9174bb7.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.

assignUser · 2024-07-27T00:12:19Z

dev/tasks/java-jars/github.yml

@@ -83,7 +83,7 @@ jobs:
          - { runs_on: ["macos-13"], arch: "x86_64"}
          - { runs_on: ["macos-14"], arch: "aarch_64" }
    env:
-      MACOSX_DEPLOYMENT_TARGET: "10.15"
+      MACOSX_DEPLOYMENT_TARGET: "14.0"


@vibhatha That's quite the bump, I am pretty sure we also want to support less recent macos.

Specify the minimum version of the target platform (e.g. macOS or iOS) on which the target binaries are to be deployed.

Ah, my bad for merging this quickly. I was too eager to get java-jars working again. Let's revert that part of the diff since it's not necessary.

I was too eager to get java-jars working again.

No worry, it's a good goal^^ We can likely bump it to 11 or even 12 (as 11 is also eol... but we don't have a support policy so...)

@assignUser this was an artifact from my previous failed attempt to fix this issue. I have not reverted that. But I will revert it shortly.

kou · 2024-07-29T02:06:55Z

dev/tasks/java-jars/github.yml

+
+          # We want to use the bundled googletest for static linking. Since
+          # both BUNDLED and brew options are enabled, it could cause a conflict
+          # when there is a version mismatch.
+          # We uninstall googletest to ensure using the bundled googletest.
+          brew uninstall googletest


Could you move this part before brew bundle --file=arrow/java/Brewfile?
Because googletest is in cpp/Brewfile not java/Brewfile.

Sure @kou I will make a follow up PR shortly.

@kou PR is here: #43462

github-actions bot added the awaiting review Awaiting review label Jul 23, 2024

llama90 reviewed Jul 23, 2024

View reviewed changes

dev/tasks/java-jars/github.yml Outdated Show resolved Hide resolved

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jul 23, 2024

vibhatha added 5 commits July 26, 2024 04:56

fix: initial check

606ee91

fix: test 1

df6e5b8

fix: adding a specific version googletest to validate the issue

834a4d4

fix: adding a specific version googletest to validate the issue

019f3d2

fix: remove brew installed version of googletest

85c1244

vibhatha force-pushed the gh-43377 branch from 51f0764 to 85c1244 Compare July 25, 2024 23:29

fix: adding comments

10c59a8

vibhatha marked this pull request as ready for review July 26, 2024 00:01

vibhatha requested review from assignUser, kou and raulcd as code owners July 26, 2024 00:01

danepitkin approved these changes Jul 26, 2024

View reviewed changes

danepitkin merged commit 9174bb7 into apache:main Jul 26, 2024
9 checks passed

danepitkin removed the awaiting committer review Awaiting committer review label Jul 26, 2024

danepitkin mentioned this pull request Jul 26, 2024

[Java][CI] Java-Jars CI is Failing with a linking error on macOS #43377

Closed

assignUser reviewed Jul 27, 2024

View reviewed changes

github-actions bot added the awaiting committer review Awaiting committer review label Jul 27, 2024

kou reviewed Jul 29, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Jul 29, 2024

GH-43377: [Java][CI] Java-Jars CI is Failing with a linking error on macOS #43385

GH-43377: [Java][CI] Java-Jars CI is Failing with a linking error on macOS #43385

Conversation

vibhatha commented Jul 23, 2024 • edited Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

vibhatha commented Jul 23, 2024

github-actions bot commented Jul 23, 2024

github-actions bot commented Jul 23, 2024

llama90 left a comment

Choose a reason for hiding this comment

llama90 commented Jul 23, 2024

vibhatha commented Jul 23, 2024

llama90 commented Jul 23, 2024

vibhatha commented Jul 23, 2024

github-actions bot commented Jul 23, 2024

vibhatha commented Jul 23, 2024

github-actions bot commented Jul 23, 2024

danepitkin commented Jul 23, 2024 • edited Loading

vibhatha commented Jul 24, 2024

kou commented Jul 24, 2024

vibhatha commented Jul 25, 2024

vibhatha commented Jul 25, 2024

vibhatha commented Jul 25, 2024

github-actions bot commented Jul 25, 2024

vibhatha commented Jul 25, 2024

github-actions bot commented Jul 25, 2024

vibhatha commented Jul 25, 2024

github-actions bot commented Jul 25, 2024

vibhatha commented Jul 25, 2024

github-actions bot commented Jul 25, 2024

vibhatha commented Jul 26, 2024

github-actions bot commented Jul 26, 2024

danepitkin left a comment

Choose a reason for hiding this comment

conbench-apache-arrow bot commented Jul 26, 2024

assignUser Jul 27, 2024 • edited Loading

Choose a reason for hiding this comment

danepitkin Jul 27, 2024

Choose a reason for hiding this comment

assignUser Jul 27, 2024

Choose a reason for hiding this comment

vibhatha Jul 29, 2024

Choose a reason for hiding this comment

kou Jul 29, 2024

Choose a reason for hiding this comment

vibhatha Jul 29, 2024

Choose a reason for hiding this comment

vibhatha Jul 29, 2024

Choose a reason for hiding this comment

vibhatha commented Jul 23, 2024 •

edited

Loading

danepitkin commented Jul 23, 2024 •

edited

Loading

assignUser Jul 27, 2024 •

edited

Loading