[Arrow] Another try at building Arrow #5425

evetion · 2022-09-04T09:03:46Z

Previous attempts at #918 and #1645. This is a long shot, but I really like to have #5423 working with GeoParquet.

A/Arrow/build_tarballs.jl

evetion · 2022-09-05T21:21:59Z

It seems xsimd and re2 are required libraries that are not available on Yggdrasil (yet), so it will try to download and compile them on the fly, loosing our toolchain context in the process.

edit: I've added a RE2 build. xsimd seems to be header only, so that should "just work".

evetion · 2022-09-11T15:57:30Z

Current state:

Windows builds failing on linking with:

[15:48:20] /opt/bin/x86_64-w64-mingw32-libgfortran5-cxx11/x86_64-w64-mingw32-g++ --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/  -Wno-noexcept-type  -fdiagnostics-color=always -O3 -DNDEBUG  -Wa,-mbig-obj -Wall -fno-semantic-interposition -mxsave  -O3 -DNDEBUG -Wl,--version-script=/workspace/srcdir/arrow-apache-arrow-9.0.0/cpp/src/arrow/symbols.map -shared -o ../../release/libarrow.dll -Wl,--out-implib,../../release/libarrow.dll.a -Wl,--major-image-version,900,--minor-image-version,0 -Wl,--whole-archive CMakeFiles/arrow_shared.dir/objects.a -Wl,--no-whole-archive @CMakeFiles/arrow_shared.dir/linklibs.rsp
[15:48:32] /opt/x86_64-w64-mingw32/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld: /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/../lib/libmsvcrt.a(lib64_libmsvcrt_os_a-wcrtomb.o): in function `wcsrtombs':
[15:48:32] /workspace/srcdir/mingw-w64-v7.0.0/mingw-w64-crt/misc/wcrtomb.c:58: multiple definition of `wcsrtombs'; /workspace/destdir/lib/libzstd.dll.a(d000591.o):(.text+0x0): first defined here
[15:48:32] /opt/x86_64-w64-mingw32/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld: CMakeFiles/arrow_shared.dir/objects.a(scalar_string_utf8.cc.obj):scalar_string_utf8.cc:(.text+0x625): undefined reference to `__imp_utf8proc_category'
[15:48:32] /opt/x86_64-w64-mingw32/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld: CMakeFiles/arrow_shared.dir/objects.a(scalar_string_utf8.cc.obj):scalar_string_utf8.cc:(.text+0x10cc): undefined reference to `__imp_utf8proc_get_property'
[15:48:32] /opt/x86_64-w64-mingw32/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld: CMakeFiles/arrow_shared.dir/objects.a(scalar_string_utf8.cc.obj):scalar_string_utf8.cc:(.text+0x1129): undefined reference to `__imp_utf8proc_category'
[15:48:32] /opt/x86_64-w64-mingw32/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld: CMakeFiles/arrow_shared.dir/objects.a(scalar_string_utf8.cc.obj):scalar_string_utf8.cc:(.text+0x1229): undefined reference to `__imp_utf8proc_get_property'
[15:48:32] /opt/x86_64-w64-mingw32/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld: CMakeFiles/arrow_shared.dir/objects.a(scalar_string_utf8.cc.obj):scalar_string_utf8.cc:(.text+0x1308): undefined reference to `__imp_utf8proc_category'
...

FreeBSD failing (and some linux musl builds) on

[15:45:25] cd /workspace/srcdir/arrow-apache-arrow-9.0.0/cpp/build_dir/src/arrow && /usr/bin/ccache /opt/bin/x86_64-unknown-freebsd12.2-libgfortran5-cxx11/x86_64-unknown-freebsd12.2-clang++ --sysroot=/opt/x86_64-unknown-freebsd12.2/x86_64-unknown-freebsd12.2/sys-root/ -DARROW_EXPORTING -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_WITH_BACKTRACE -DARROW_WITH_BZ2 -DARROW_WITH_LZ4 -DARROW_WITH_RE2 -DARROW_WITH_SNAPPY -DARROW_WITH_TIMING_TESTS -DARROW_WITH_UTF8PROC -DARROW_WITH_ZLIB -DARROW_WITH_ZSTD -DBOOST_ALL_NO_LIB -DURI_STATIC_BUILD -I/workspace/srcdir/arrow-apache-arrow-9.0.0/cpp/build_dir/src -I/workspace/srcdir/arrow-apache-arrow-9.0.0/cpp/src -I/workspace/srcdir/arrow-apache-arrow-9.0.0/cpp/src/generated -isystem /workspace/srcdir/arrow-apache-arrow-9.0.0/cpp/thirdparty/flatbuffers/include -isystem /workspace/srcdir/arrow-apache-arrow-9.0.0/cpp/thirdparty/hadoop/include -isystem /workspace/srcdir/arrow-apache-arrow-9.0.0/cpp/build_dir/xsimd_ep/src/xsimd_ep-install/include -Qunused-arguments -fcolor-diagnostics -O3 -DNDEBUG  -Wall -Wno-unknown-warning-option -Wno-pass-failed  -O3 -DNDEBUG -fPIC -pthread -std=c++11 -MD -MT src/arrow/CMakeFiles/arrow_objlib.dir/compute/exec/sink_node.cc.o -MF CMakeFiles/arrow_objlib.dir/compute/exec/sink_node.cc.o.d -o CMakeFiles/arrow_objlib.dir/compute/exec/sink_node.cc.o -c /workspace/srcdir/arrow-apache-arrow-9.0.0/cpp/src/arrow/compute/exec/sink_node.cc
[15:45:27] /workspace/srcdir/arrow-apache-arrow-9.0.0/cpp/src/arrow/util/cpu_info.cc:303:11: error: use of undeclared identifier '_SC_LEVEL1_DCACHE_SIZE'
[15:45:27]           _SC_LEVEL1_DCACHE_SIZE,
[15:45:27]           ^
...

macOS and some Linux builds are failing on trying to configure jemalloc, so it wasn't found.

Some Linux arm builds failing on finding BZip2.

giordano · 2022-09-11T16:03:35Z

If you want to make everybody's life easier, when you show linking errors you also want to show the compiler invocation line. Because the undefined reference to error means that the linker can't find the mentioned symbol in any of the linked libraries, which often simply means that the library which is supposed to implement the missing symbol wasn't passed as argument. But if I don't see the compiler invocation I can't tell you "see, that's the problem".

evetion · 2022-09-11T16:16:55Z

If you want to make everybody's life easier, when you show linking errors you also want to show the compiler invocation line.

Yep 👍🏻, thanks for the feedback, updated my post and linked to the logs.

giordano · 2022-09-11T16:19:52Z

Yeah, the compiler invocation

[15:48:20] /opt/bin/x86_64-w64-mingw32-libgfortran5-cxx11/x86_64-w64-mingw32-g++ --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/  -Wno-noexcept-type  -fdiagnostics-color=always -O3 -DNDEBUG  -Wa,-mbig-obj -Wall -fno-semantic-interposition -mxsave  -O3 -DNDEBUG -Wl,--version-script=/workspace/srcdir/arrow-apache-arrow-9.0.0/cpp/src/arrow/symbols.map -shared -o ../../release/libarrow.dll -Wl,--out-implib,../../release/libarrow.dll.a -Wl,--major-image-version,900,--minor-image-version,0 -Wl,--whole-archive CMakeFiles/arrow_shared.dir/objects.a -Wl,--no-whole-archive @CMakeFiles/arrow_shared.dir/linklibs.rsp

doesn't link utf8proc. Don't ask me how on earth this is supposed to work at all (unless they, wrongly, assume to do static linking everywhere).

evetion · 2022-10-30T14:34:27Z

Fixed previous problems by just excluding zstd and utf8proc. Also disabled building utilities. Upgraded to Arrow v10.

giordano · 2022-10-30T15:10:44Z

A/Arrow/build_tarballs.jl

+# CMake is doubling the suffixes...
+if [[ "${target}" == *-mingw32 ]]; then
+    ln -s ${prefix}/lib/libthrift.dll.a ${prefix}/lib/libthrift.a.dll.a
+    ln -s ${prefix}/lib/libutf8proc.a ${prefix}/lib/libutf8proc.dll.a.a
+fi


Sounds like something to fix in CMake, rather than creating files with wrong extensions which go into the tarball (at very least you should delete them at the end)

I'm still of the opinion this should be fixed in CMake, not worked around in this way. If that's of any help, you want to look at to CMAKE_FIND_LIBRARY_SUFFIXES.

Agreed, I will look into it fixing/upstreaming (also the other bugs).

This is now removed, turned out to be unnecessary in Arrow v10.

A/Arrow/build_tarballs.jl

A/Arrow/bundled/patches/cxxflags.patch

A/Arrow/bundled/patches/thrift.patch

A/Arrow/build_tarballs.jl

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

A/Arrow/build_tarballs.jl

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

giordano · 2022-11-01T21:17:33Z

A/Arrow/build_tarballs.jl

+# CMake is doubling the suffixes...
+if [[ "${target}" == *-mingw32 ]]; then
+    ln -s ${prefix}/lib/libthrift.dll.a ${prefix}/lib/libthrift.a.dll.a
+    ln -s ${prefix}/lib/libutf8proc.a ${prefix}/lib/libutf8proc.dll.a.a
+fi


I'm still of the opinion this should be fixed in CMake, not worked around in this way. If that's of any help, you want to look at to CMAKE_FIND_LIBRARY_SUFFIXES.

A/Arrow/build_tarballs.jl

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

evetion · 2022-11-06T11:10:25Z

Made the following issues:

ARMv6 support https://issues.apache.org/jira/browse/ARROW-18255
Libthrift IMPORT_LIB https://issues.apache.org/jira/browse/ARROW-18256

A/Arrow/build_tarballs.jl

evetion added 2 commits September 4, 2022 10:48

Another try at building Arrow.

79f4b6b

Add compression deps.

1f072ce

giordano added the long shot 🏹 This is going to be fun label Sep 4, 2022

Add jemalloc dep.

6739697

giordano reviewed Sep 4, 2022

View reviewed changes

A/Arrow/build_tarballs.jl Outdated Show resolved Hide resolved

Set compat for deps and try to use jemalloc.

0830ba4

evetion mentioned this pull request Sep 4, 2022

[Thrift] Build thrift library with cmake. #5427

Merged

evetion added 2 commits September 5, 2022 12:03

Add Thrift as dep.

1e6bc39

Use only system libs.

a50ed96

visr reviewed Sep 5, 2022

View reviewed changes

A/Arrow/build_tarballs.jl Outdated Show resolved Hide resolved

Let's try a hack.

00ec791

evetion mentioned this pull request Sep 7, 2022

[RE2] Add a RE2 build #5440

Merged

evetion added 3 commits September 11, 2022 14:57

Use RE2_jll.

26e0424

Use gcc v6 to get (partial) c++17.

3ea047d

Use gcc v8 to get full c++17 support.

bcb9b13

Disable building jemalloc and skip Bzip2 support when needed.

f222a83

giordano mentioned this pull request Sep 15, 2022

[SPEX] Add builder #5504

Merged

evetion added 5 commits October 30, 2022 11:50

Simplify build considerably.

75b9ba2

Upgrade to Arrow 10 to support ARM better.

a887249

Patch support for armv6, disable older macos checks.

4aeaf24

Fix passing CXX flags for macos.

2042ec8

Cleanup.

e075b72

evetion marked this pull request as ready for review October 30, 2022 14:32

giordano reviewed Oct 30, 2022

View reviewed changes

A/Arrow/build_tarballs.jl Show resolved Hide resolved

evetion and others added 2 commits October 30, 2022 20:49

Apply suggestions from code review

c6b2a81

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

Remove temporary symlinks. Always build with BZ2.

98b563c

giordano reviewed Oct 30, 2022

View reviewed changes

A/Arrow/build_tarballs.jl Outdated Show resolved Hide resolved

Update A/Arrow/build_tarballs.jl

23f7375

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

giordano reviewed Nov 1, 2022

View reviewed changes

Apply suggestions from code review

eff59a0

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

Removed unneeded extension patching on mingw.

c063814

giordano reviewed Nov 6, 2022

View reviewed changes

A/Arrow/build_tarballs.jl Outdated Show resolved Hide resolved

Update A/Arrow/build_tarballs.jl

0941ebf

giordano enabled auto-merge (squash) November 6, 2022 19:44

giordano mentioned this pull request Nov 6, 2022

Add Arrow builder #918

Closed

giordano merged commit 5f1b155 into JuliaPackaging:master Nov 6, 2022

visr mentioned this pull request Nov 11, 2022

[GDAL] build 3.6.0 and add Arrow & Parquet #5848

Merged

asfimport mentioned this pull request Nov 18, 2022

[C++][Windows] External/shared libthrift requires IMPORTED_IMPLIB in FindThrift.cmake apache/arrow#33437

Closed

giordano mentioned this pull request Jan 22, 2023

[OSRM] New Binary #6106

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Arrow] Another try at building Arrow #5425

[Arrow] Another try at building Arrow #5425

evetion commented Sep 4, 2022

evetion commented Sep 5, 2022 •

edited

Loading

evetion commented Sep 11, 2022 •

edited

Loading

giordano commented Sep 11, 2022 •

edited

Loading

evetion commented Sep 11, 2022

giordano commented Sep 11, 2022

evetion commented Oct 30, 2022

giordano Oct 30, 2022

giordano Nov 1, 2022

evetion Nov 2, 2022

evetion Nov 6, 2022

giordano Nov 1, 2022

evetion commented Nov 6, 2022

[Arrow] Another try at building Arrow #5425

[Arrow] Another try at building Arrow #5425

Conversation

evetion commented Sep 4, 2022

evetion commented Sep 5, 2022 • edited Loading

evetion commented Sep 11, 2022 • edited Loading

giordano commented Sep 11, 2022 • edited Loading

evetion commented Sep 11, 2022

giordano commented Sep 11, 2022

evetion commented Oct 30, 2022

giordano Oct 30, 2022

Choose a reason for hiding this comment

giordano Nov 1, 2022

Choose a reason for hiding this comment

evetion Nov 2, 2022

Choose a reason for hiding this comment

evetion Nov 6, 2022

Choose a reason for hiding this comment

giordano Nov 1, 2022

Choose a reason for hiding this comment

evetion commented Nov 6, 2022

evetion commented Sep 5, 2022 •

edited

Loading

evetion commented Sep 11, 2022 •

edited

Loading

giordano commented Sep 11, 2022 •

edited

Loading