Ensure that the software supports big-endian hardware

[lemire]

Currently, the code is only tested on little-endian hardware. We would need to run tests and to flip byte order (to preserve interoperability) on big-endian hardware.

[marshallpierce]

Perhaps qemu would be a good way to do this? I used https://translatedcode.wordpress.com/2016/11/03/installing-debian-on-qemus-32-bit-arm-virt-board/ a while back when I needed to test something on ARM (and now there is an updated 64-bit version). I'm not sure how good qemu's support for SIMD extensions is, but even if performance isn't representative perhaps we could at least test for correctness of a hypothetical NEON implementation.

[lemire]

The ARM-based systems I work with are little-endian by default. iOS is certainly little endian. I actually don't have access to a little-endian system.

Please see this other issue which is more relevant:

Port to ARM NEON (aarch64)
#6

[marshallpierce]

I should have been more clear -- I didn't mean that qemu was only useful for ARM, though I definitely did give that impression... Anyway qemu supports many architectures.

[lemire]

Yes. QEMU is fantastic software.

From a practical point of view, I am primarily interested in x64 and Aarch64 architectures, and I think that they should be given priority. Such systems are, by default, little endian.

[KWillets]

One note about shuffles is that we can switch endianness in u32 lanes by setting index ^= 3. So if we were eg retrieving index 0 little-endian we get 3 big-endian. This reverses bytes for any 4-aligned u32.

[outpaddling]

Ran it on an iMac G5 running FreeBSD 13.0, clang 11.0.1:

===>  Testing for streamvbyte-0.4.1
cd /usr/ports/wip/streamvbyte/work/.build && ./unit
[streamvbyte_decode] code is buggy gap=729
*** Error code 255

The test passes on amd64 and aarch64, both also running FreeBSD 13.

Cheers,

Jason

[lemire]

@outpaddling

That's a power processor, right? Do we support power processors?

I am impressed that it is building at all... ?

Anyhow, yeah, it is not going to work right now on big endian systems. I'll add a note to the README.

[lemire]

@outpaddling The README said...

It assumes a recent Intel processor (e.g., haswell or better, though we provide runtime dispatching for compatibility with legacy systems) or an ARM processor with NEON instructions (which is almost all of them except for the tiny cores).

I added the following to be explicit:

Big-endian processors are unsupported at this time.

People who need big-endian POWER support are invited to give us a hand.

[outpaddling]

As the POWER architecture is also moving to LE, not sure anyone will be very motivated at this point. Just gave it a try because I could. If anyone is interested enough to work on the code, I'll be happy to test patches on my iMac G5.

[lemire]

@outpaddling Thanks.