Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[C++][CI] OpenSSL link error in Gandiva on macOS #36329

Closed
pitrou opened this issue Jun 27, 2023 · 2 comments · Fixed by #36336
Closed

[C++][CI] OpenSSL link error in Gandiva on macOS #36329

pitrou opened this issue Jun 27, 2023 · 2 comments · Fixed by #36336
Assignees
@kou
Labels
Component: C++ Component: Continuous Integration Priority: Blocker Marks a blocker for the release Type: bug
Milestone
13.0.0

Comments

@pitrou
Copy link
Member

pitrou commented Jun 27, 2023

Describe the bug, including details regarding any error messages, version, and platform.

As seen on CI:

Undefined symbols for architecture x86_64:
  "_EVP_MD_get_size", referenced from:
      gandiva::gdv_hash_using_openssl(long long, void const*, unsigned long, evp_md_st const*, unsigned int, int*) in libgandiva.a(hash_utils.cc.o)
ld: symbol(s) not found for architecture x86_64

https://github.com/apache/arrow/actions/runs/5390688418/jobs/9786417735#step:9:1716

Also, it would be nice if the CMake configure logs shows the architecture being targeted (perhaps CMAKE_SYSTEM_PROCESSOR?).

Component(s)

C++, Continuous Integration
@pitrou
Copy link
Member Author

pitrou commented Jun 27, 2023

cc @kou

@kou
Copy link
Member

kou commented Jun 27, 2023

Ah, sorry. I didn't take a look at this failure.

It's caused by mixing OpenSSL 1.1 and 3.0.
In this case, OpenSSL 3.0's include headers (they have #define EVP_MD_size EVP_MD_get_size) are used and OpenSSL 1.1's libraries (they provides EVP_MD_size not EVP_MD_get_size) are used.

kou added a commit to kou/arrow that referenced this issue Jun 28, 2023
@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
fc0a16e 
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.

If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.

This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.

Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
@github-actions github-actions bot mentioned this issue Jun 28, 2023
Merged
laurentgo pushed a commit to laurentgo/arrow that referenced this issue Jun 28, 2023
@kou @laurentgo
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
ad2fc1f 
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.

If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.

This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.

Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
laurentgo pushed a commit to laurentgo/arrow that referenced this issue Jun 28, 2023
@kou @laurentgo
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
8a23c95 
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.

If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.

This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.

Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
@raulcd raulcd added the Priority: Blocker Marks a blocker for the release label Jun 28, 2023
laurentgo pushed a commit to laurentgo/arrow that referenced this issue Jun 28, 2023
@kou @laurentgo
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
c385adb 
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.

If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.

This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.

Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
laurentgo pushed a commit to laurentgo/arrow that referenced this issue Jun 28, 2023
@kou @laurentgo
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
bd6c485 
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.

If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.

This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.

Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
laurentgo pushed a commit to laurentgo/arrow that referenced this issue Jun 28, 2023
@kou @laurentgo
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
c0a63e9 
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.

If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.

This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.

Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
laurentgo pushed a commit to laurentgo/arrow that referenced this issue Jun 28, 2023
@kou @laurentgo
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
11e2814 
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.

If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.

This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.

Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
@bkietz bkietz mentioned this issue Jun 29, 2023
Merged
kou added a commit that referenced this issue Jun 30, 2023
@kou
GH-36329: [C++][CI] Use OpenSSL 3 on macOS (#36336)
9d92ed4 
### Rationale for this change

GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.

If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.

### What changes are included in this PR?

This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.

Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* Closes: #36329

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@kou kou added this to the 13.0.0 milestone Jun 30, 2023
lriggs pushed a commit to lriggs/arrow that referenced this issue Jul 19, 2023
@kou @lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
8c582cd 
### Rationale for this change

GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.

If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.

### What changes are included in this PR?

This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.

Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* Closes: apache#36329

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
lriggs pushed a commit to dremio/arrow that referenced this issue Jul 21, 2023
@kou @lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
6eb8141 
### Rationale for this change

GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.

If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.

### What changes are included in this PR?

This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.

Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* Closes: apache#36329

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@xxlaykxx xxlaykxx mentioned this issue Jul 30, 2023
Merged
xxlaykxx added a commit to dremio/arrow that referenced this issue Jul 30, 2023
@xxlaykxx @kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336) (#41)
cfe29f2 
### Rationale for this change

GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.

If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.

### What changes are included in this PR?

This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.

Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* Closes: apache#36329

Authored-by: Sutou Kouhei <kou@clear-code.com>

Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kou kou

Labels
Component: C++ Component: Continuous Integration Priority: Blocker Marks a blocker for the release Type: bug
Projects
None yet
Milestone
13.0.0
Development

Successfully merging a pull request may close this issue.

GH-36329: [C++][CI] Use OpenSSL 3 on macOS kou/arrow
3 participants
@kou @raulcd @pitrou