Failed to build for arm

[webigorkiev]

Failed to build for arm64 debian 10.
Apparently the FastPFOR library only supports the x64 platform.
Please add build from source instructions to the documentation.

You might be interested in https://github.com/powturbo/TurboPFor-Integer-Compression

OS/Compiler (64 bits):
Windows: MinGW-w64 makefile
Windows: Visual c++ (>=VS2008) - makefile.vs (for nmake)
Windows: Visual Studio project file - vs/vs2017 - Thanks to PavelP
Linux amd64: GNU GCC (>=4.6)
Linux amd64: Clang (>=3.2)
Linux arm64: 64 bits aarch64 ARMv8: gcc (>=6.3)
Linux arm64: 64 bits aarch64 ARMv8: clang
MaxOS: XCode (>=9)
PowerPC ppc64le (incl. SIMD): gcc (>=8.0)

Thanks.

[uper]

on Macos M1,Failed to build

[sanikolaev]

@webigorkiev @uper

The columnar library doesn't support ARM yet. Please explain why it's important for you to build/use the library on Macos M1 or other ARM hardware. It may help us to set priorities properly.

[uper]

@sanikolaev for development

[sanikolaev]

@uper do you mean you'd like to modify the columnar library, but can't since you can't build it on arm?

[webigorkiev]

@webigorkiev @uper

The columnar library doesn't support ARM yet. Please explain why it's important for you to build/use the library on Macos M1 or other ARM hardware. It may help us to set priorities properly.

ARM is a fairly wide range of processors. The entire graviton family on AWS. M1 at Apple. Apple also announced the transition to arm64 for all its future products.
So far, we are just looking at columnar. But, it can be widely used in analytics and search systems for logs, big data.
In any case, ARM64 is the future, and while the project does not yet depend 100% on SSE and AVX instructions, it would be very useful to add support for ARM. (And arm neon)
Thanks.

[sanikolaev]

Unfortunately we can't use TurboPFOR due to the licenses incompatibility issues. Manticore Columnar Library is based on Apache 2.0 while TurboPFOR is GPLv2

[tangxinfa]

Maybe we can use streamvbyte, which
use Apache 2.0 License, and support arm64.

It also suggested by the FastPFor library
author, see fast-pack/FastPFor#63 (comment)

[sanikolaev]

We do use streamvbyte for 32-bit integers, but libfastpfor has other codecs that are also important and can't be built on ARM64. Best of all would be to port libfastpfor to ARM64.

[tangxinfa]

It's not too difficult to build columnar on raspberry arm64 by use
simde library.

I build it and run successfully on my raspberry arm64 device, indexed 1G text
files and query on it without errors.

Install simde library first, then use the following fork to build columnar module:

https://github.com/tangxinfa/FastPFor/tree/fix-aarch64
https://github.com/tangxinfa/columnar/tree/fix-aarch64

My fork just did a minor modification on the original version.

CAUTION

My fork of FastPFor library can successfully compile on arm64, but `make
check` will fill on some test cases. We should investigate it if the fail parts
used in columnar module.

[sanikolaev]

will fail on some test cases

Can you please elaborate more on this? What tests are failed?

[tangxinfa]

I did the following to build and run FastPFor test cases on raspberry pi arm64 device.

1, Get the code from my fork

git clone https://github.com/tangxinfa/FastPFor.git -b fix-aarch64

2, Build and run test cases

cd FastPFor && mkdir build
cmake -B build
make -C build check

Unit tests fail log:

...
Scanning dependencies of target check
make[3]: Leaving directory '/home/tangxinfa/Examples/FastPFor/build'
make[3]: Entering directory '/home/tangxinfa/Examples/FastPFor/build'
testing... b = 0
testing... b = 1
recovered[36] = 0
data[36] = 1
recovered[37] = 0
data[37] = 1
recovered[38] = 0
data[38] = 1
recovered[39] = 0
data[39] = 1
recovered[40] = 0
data[40] = 1
recovered[41] = 0
data[41] = 1
recovered[42] = 0
data[42] = 1
recovered[43] = 0
data[43] = 1
recovered[44] = 0
data[44] = 1
recovered[45] = 0
data[45] = 1
recovered[46] = 0
data[46] = 1
recovered[47] = 0
data[47] = 1
recovered[48] = 0
data[48] = 1
recovered[49] = 0
data[49] = 1
recovered[50] = 0
data[50] = 1
recovered[51] = 0
data[51] = 1
SIMDBinaryPacking+VariableByte
128
terminate called after throwing an instance of 'std::logic_error'
  what():  we have a bug
make[3]: *** [CMakeFiles/check.dir/build.make:57: CMakeFiles/check] Aborted
make[3]: Leaving directory '/home/tangxinfa/Examples/FastPFor/build'
make[2]: *** [CMakeFiles/Makefile2:212: CMakeFiles/check.dir/all] Error 2
make[2]: Leaving directory '/home/tangxinfa/Examples/FastPFor/build'
make[1]: *** [CMakeFiles/Makefile2:219: CMakeFiles/check.dir/rule] Error 2
make[1]: Leaving directory '/home/tangxinfa/Examples/FastPFor/build'
make: *** [Makefile:201: check] Error 2
make: Leaving directory '/home/tangxinfa/Examples/FastPFor/build'