1.11.0 test failure on big endian architectures

[sebastic]

The Debian package build for 1.11.0 failed on s390x due to test failures:

123/313 Test #123: export-spaten ....................................***Failed    0.02 sec
Executing: /<<PKGBUILDDIR>>/obj-s390x-linux-gnu/src/osmium export -f spaten export/input.osm
Executing: /usr/bin/cmake -E compare_files /<<PKGBUILDDIR>>/test/export/output.spaten /<<PKGBUILDDIR>>/obj-s390x-linux-gnu/test/export/cmd-output-spaten
Files "/<<PKGBUILDDIR>>/test/export/output.spaten" to "/<<PKGBUILDDIR>>/obj-s390x-linux-gnu/test/export/cmd-output-spaten" are different.
CMake Error at /<<PKGBUILDDIR>>/cmake/run_test_compare_output.cmake:85 (message):
  Test output does not match
  '/<<PKGBUILDDIR>>/test/export/output.spaten'.
  Output is in
  '/<<PKGBUILDDIR>>/obj-s390x-linux-gnu/test/export/cmd-output-spaten'.

...
Executing: /<<PKGBUILDDIR>>/obj-s390x-linux-gnu/src/osmium export -f spaten --keep-untagged export/input.osm
Executing: /usr/bin/cmake -E compare_files /<<PKGBUILDDIR>>/test/export/output-untagged.spaten /<<PKGBUILDDIR>>/obj-s390x-linux-gnu/test/export/cmd-output-spaten-untagged
Files "/<<PKGBUILDDIR>>/test/export/output-untagged.spaten" to "/<<PKGBUILDDIR>>/obj-s390x-linux-gnu/test/export/cmd-output-spaten-untagged" are different.
CMake Error at /<<PKGBUILDDIR>>/cmake/run_test_compare_output.cmake:85 (message):
  Test output does not match
  '/<<PKGBUILDDIR>>/test/export/output-untagged.spaten'.
  Output is in
  '/<<PKGBUILDDIR>>/obj-s390x-linux-gnu/test/export/cmd-output-spaten-untagged'.

Full buildlogs: s390x

hexdump diffs:

--- ./test/export/output-untagged.spaten.hexdump        2019-09-17 06:33:25.318672996 +0000
+++ ./obj-s390x-linux-gnu/test/export/cmd-output-spaten-untagged.hexdump        2019-09-17 06:35:04.258672996 +0000
@@ -1,34 +1,34 @@
 0000000 5350 4154 0000 0000 f601 0000 0000 0000
-0000010 121b 0801 1000 1a15 0101 0000 0000 0000
-0000020 0000 00f0 3f00 0000 0000 00f0 3f12 1b08
-0000030 0110 001a 1501 0100 0000 0000 0000 0000
-0000040 f03f 0000 0000 0000 0040 121b 0801 1000
-0000050 1a15 0101 0000 0000 0000 0000 00f0 3f00
-0000060 0000 0000 0008 4012 1b08 0110 001a 1501
-0000070 0100 0000 0000 0000 0000 f03f 0000 0000
-0000080 0000 1040 1232 0801 1000 1a15 0101 0000
-0000090 0000 0000 0000 0000 4000 0000 0000 00f8
-00000a0 3f42 150a 0761 6d65 6e69 7479 1208 706f
+0000010 121b 0801 1000 1a15 0000 0000 013f f000
+0000020 0000 0000 003f f000 0000 0000 0012 1b08
+0000030 0110 001a 1500 0000 0001 3ff0 0000 0000
+0000040 0000 4000 0000 0000 0000 121b 0801 1000
+0000050 1a15 0000 0000 013f f000 0000 0000 0040
+0000060 0800 0000 0000 0012 1b08 0110 001a 1500
+0000070 0000 0001 3ff0 0000 0000 0000 4010 0000
+0000080 0000 0000 1232 0801 1000 1a15 0000 0000
+0000090 0140 0000 0000 0000 003f f800 0000 0000
+00000a0 0042 150a 0761 6d65 6e69 7479 1208 706f
 00000b0 7374 5f62 6f78 1800 1253 0802 1000 1a39
-00000c0 0102 0000 0003 0000 0000 0000 0000 00f0
-00000d0 3f00 0000 0000 00f0 3f00 0000 0000 00f0
-00000e0 3f00 0000 0000 0000 4000 0000 0000 00f0
-00000f0 3f00 0000 0000 0008 4042 120a 0768 6967
+00000c0 0000 0000 0200 0000 033f f000 0000 0000
+00000d0 003f f000 0000 0000 003f f000 0000 0000
+00000e0 0040 0000 0000 0000 003f f000 0000 0000
+00000f0 0040 0800 0000 0000 0042 120a 0768 6967
 0000100 6877 6179 1205 7472 6163 6b18 0012 5308
-0000110 0210 001a 3901 0200 0000 0300 0000 0000
-0000120 0000 0000 f03f 0000 0000 0000 f03f 0000
-0000130 0000 0000 f03f 0000 0000 0000 0040 0000
-0000140 0000 0000 0040 0000 0000 0000 f83f 4212
+0000110 0210 001a 3900 0000 0002 0000 0003 3ff0
+0000120 0000 0000 0000 3ff0 0000 0000 0000 3ff0
+0000130 0000 0000 0000 4000 0000 0000 0000 4000
+0000140 0000 0000 0000 3ff8 0000 0000 0000 4212
 0000150 0a07 6261 7272 6965 7212 0566 656e 6365
-0000160 1800 122f 0802 1000 1a29 0102 0000 0002
-0000170 0000 0000 0000 0000 0000 4000 0000 0000
-0000180 00f8 3f00 0000 0000 00f0 3f00 0000 0000
-0000190 00f0 3f12 7108 0310 001a 5601 0600 0000
-00001a0 0100 0000 0103 0000 0001 0000 0004 0000
-00001b0 0000 0000 0000 00f0 3f00 0000 0000 00f0
-00001c0 3f00 0000 0000 0000 4000 0000 0000 00f8
-00001d0 3f00 0000 0000 00f0 3f00 0000 0000 0000
-00001e0 4000 0000 0000 00f0 3f00 0000 0000 00f0
-00001f0 3f42 130a 076c 616e 6475 7365 1206 666f
+0000160 1800 122f 0802 1000 1a29 0000 0000 0200
+0000170 0000 0240 0000 0000 0000 003f f800 0000
+0000180 0000 003f f000 0000 0000 003f f000 0000
+0000190 0000 0012 7108 0310 001a 5600 0000 0006
+00001a0 0000 0001 0000 0000 0300 0000 0100 0000
+00001b0 043f f000 0000 0000 003f f000 0000 0000
+00001c0 0040 0000 0000 0000 003f f800 0000 0000
+00001d0 003f f000 0000 0000 0040 0000 0000 0000
+00001e0 003f f000 0000 0000 003f f000 0000 0000
+00001f0 0042 130a 076c 616e 6475 7365 1206 666f
 0000200 7265 7374 1800                         
 0000206

--- ./test/export/output.spaten.hexdump 2019-09-17 06:36:14.846672996 +0000
+++ ./obj-s390x-linux-gnu/test/export/cmd-output-spaten.hexdump 2019-09-17 06:36:40.622672068 +0000
@@ -1,24 +1,24 @@
 0000000 5350 4154 0000 0000 5101 0000 0000 0000
-0000010 1232 0801 1000 1a15 0101 0000 0000 0000
-0000020 0000 0000 4000 0000 0000 00f8 3f42 150a
+0000010 1232 0801 1000 1a15 0000 0000 0140 0000
+0000020 0000 0000 003f f800 0000 0000 0042 150a
 0000030 0761 6d65 6e69 7479 1208 706f 7374 5f62
-0000040 6f78 1800 1253 0802 1000 1a39 0102 0000
-0000050 0003 0000 0000 0000 0000 00f0 3f00 0000
-0000060 0000 00f0 3f00 0000 0000 00f0 3f00 0000
-0000070 0000 0000 4000 0000 0000 00f0 3f00 0000
-0000080 0000 0008 4042 120a 0768 6967 6877 6179
+0000040 6f78 1800 1253 0802 1000 1a39 0000 0000
+0000050 0200 0000 033f f000 0000 0000 003f f000
+0000060 0000 0000 003f f000 0000 0000 0040 0000
+0000070 0000 0000 003f f000 0000 0000 0040 0800
+0000080 0000 0000 0042 120a 0768 6967 6877 6179
 0000090 1205 7472 6163 6b18 0012 5308 0210 001a
-00000a0 3901 0200 0000 0300 0000 0000 0000 0000
-00000b0 f03f 0000 0000 0000 f03f 0000 0000 0000
-00000c0 f03f 0000 0000 0000 0040 0000 0000 0000
-00000d0 0040 0000 0000 0000 f83f 4212 0a07 6261
+00000a0 3900 0000 0002 0000 0003 3ff0 0000 0000
+00000b0 0000 3ff0 0000 0000 0000 3ff0 0000 0000
+00000c0 0000 4000 0000 0000 0000 4000 0000 0000
+00000d0 0000 3ff8 0000 0000 0000 4212 0a07 6261
 00000e0 7272 6965 7212 0566 656e 6365 1800 1271
-00000f0 0803 1000 1a56 0106 0000 0001 0000 0001
-0000100 0300 0000 0100 0000 0400 0000 0000 0000
-0000110 0000 f03f 0000 0000 0000 f03f 0000 0000
-0000120 0000 0040 0000 0000 0000 f83f 0000 0000
-0000130 0000 f03f 0000 0000 0000 0040 0000 0000
-0000140 0000 f03f 0000 0000 0000 f03f 4213 0a07
+00000f0 0803 1000 1a56 0000 0000 0600 0000 0100
+0000100 0000 0003 0000 0001 0000 0004 3ff0 0000
+0000110 0000 0000 3ff0 0000 0000 0000 4000 0000
+0000120 0000 0000 3ff8 0000 0000 0000 3ff0 0000
+0000130 0000 0000 4000 0000 0000 0000 3ff0 0000
+0000140 0000 0000 3ff0 0000 0000 0000 4213 0a07
 0000150 6c61 6e64 7573 6512 0666 6f72 6573 7418
 0000160 0000                                   
 0000161

[joto]

Looks like the "Spaten" export depends on byte-order. Not hugely important, but we have to look into this.

@thomersch

[sebastic]

Yes, the docs state:

Please note that unless stated otherwise encoding is assumed to be little endian.

Converting the data to LE should resolve the issue, I think.

[tomhughes]

The problem as you say is that spaten includes WKB data which can be of either endianness, with the first byte being 0 for LE and 1 for BE and the rest of the data then being byte swapped - see header and str_push in libosmium's wkb.hpp for the actual source.

So I guess we need to have two sets of valid output or something and allow either, or the correct one for the current host...

[joto]

There are several issues here:

1. As @sebastic mentions, the Spaten file format says that all data is little endian, but, as @tomhughes mentions, WKB is used in Spaten and that can be either. So the question is: Does the WKB in Spaten always have to be little endian or do we look into the WKB itself to decide which one it is?
2. Libosmium is used here to create the WKB and the code in libosmium only supports writing WKB in the native format for the given platform. So if we decide to always write the WKB in little endian, we need to extend the WKB code in libosmium (or use something else).
3. The simplistic tests do a binary comparison, which isn't good enough here.

[thomersch]

Thanks everyone for the analysis.

At the moment I would tend to restrict WKB to little endian in the specification, which would simplify decoder implementation. As a reference point: PostGIS defaults to server byte order, but is able export either.

The simplistic tests do a binary comparison, which isn't good enough here.

True, the tests are very primitive, but I am not sure how to test output without a dedicated decoder. Do you have a suggestion how to solve this, @joto?

[sebastic]

The issue is still present in 1.11.1:

https://buildd.debian.org/status/fetch.php?pkg=osmium-tool&arch=s390x&ver=1.11.1-1&stamp=1574312870&file=log

[tomhughes]

Remove the spaten test did fix this case, but adding the pg format export test introduced a different version as that also include WKB data that is endian specific.

[sebastic]

Seeing the same test failures on s390x for the Debian package:

131: Test command: /usr/bin/cmake "-D" "cmd:FILEPATH=/home/sebastic/git/pkg-grass/osmium-tool/obj-s390x-linux-gnu/src/osmium export -f pg export/input.osm" "-D" "dir:PATH=/home/sebastic/git/pkg-grass/osmium-tool/test" "-D" "reference:FILEPATH=/home/sebastic/git/pkg-grass/osmium-tool/test/export/output.pg" "-D" "output:FILEPATH=/home/sebastic/git/pkg-grass/osmium-tool/obj-s390x-linux-gnu/test/export/cmd-output-pg" "-D" "return_code=0" "-P" "/home/sebastic/git/pkg-grass/osmium-tool/cmake/run_test_compare_output.cmake"
131: Working Directory: /home/sebastic/git/pkg-grass/osmium-tool/obj-s390x-linux-gnu/test/export
131: Test timeout computed to be: 10000000
131: Executing: /home/sebastic/git/pkg-grass/osmium-tool/obj-s390x-linux-gnu/src/osmium export -f pg export/input.osm
131: Executing: /usr/bin/cmake;-E;compare_files;/home/sebastic/git/pkg-grass/osmium-tool/test/export/output.pg;/home/sebastic/git/pkg-grass/osmium-tool/obj-s390x-linux-gnu/test/export/cmd-output-pg
131: CMake Error at /home/sebastic/git/pkg-grass/osmium-tool/cmake/run_test_compare_output.cmake:84 (message):
131:   Test output does not match
131:   '/home/sebastic/git/pkg-grass/osmium-tool/test/export/output.pg'.  Output
131:   is in
131:   '/home/sebastic/git/pkg-grass/osmium-tool/obj-s390x-linux-gnu/test/export/cmd-output-pg'.
131:   Result: 1.
131: 
131: 
131/356 Test #131: export-pg ........................................***Failed    0.03 sec
        Start 154: export-pg-untagged
[...]
154: Test command: /usr/bin/cmake "-D" "cmd:FILEPATH=/home/sebastic/git/pkg-grass/osmium-tool/obj-s390x-linux-gnu/src/osmium export -f pg --keep-untagged export/input.osm" "-D" "dir:PATH=/home/sebastic/git/pkg-grass/osmium-tool/test" "-D" "reference:FILEPATH=/home/sebastic/git/pkg-grass/osmium-tool/test/export/output-untagged.pg" "-D" "output:FILEPATH=/home/sebastic/git/pkg-grass/osmium-tool/obj-s390x-linux-gnu/test/export/cmd-output-pg-untagged" "-D" "return_code=0" "-P" "/home/sebastic/git/pkg-grass/osmium-tool/cmake/run_test_compare_output.cmake"
154: Working Directory: /home/sebastic/git/pkg-grass/osmium-tool/obj-s390x-linux-gnu/test/export
154: Test timeout computed to be: 10000000
154: Executing: /home/sebastic/git/pkg-grass/osmium-tool/obj-s390x-linux-gnu/src/osmium export -f pg --keep-untagged export/input.osm
154: Executing: /usr/bin/cmake;-E;compare_files;/home/sebastic/git/pkg-grass/osmium-tool/test/export/output-untagged.pg;/home/sebastic/git/pkg-grass/osmium-tool/obj-s390x-linux-gnu/test/export/cmd-output-pg-untagged
154: CMake Error at /home/sebastic/git/pkg-grass/osmium-tool/cmake/run_test_compare_output.cmake:84 (message):
154:   Test output does not match
154:   '/home/sebastic/git/pkg-grass/osmium-tool/test/export/output-untagged.pg'.
154:   Output is in
154:   '/home/sebastic/git/pkg-grass/osmium-tool/obj-s390x-linux-gnu/test/export/cmd-output-pg-untagged'.
154:   Result: 1.
154: 
154: 
154/356 Test #154: export-pg-untagged ...............................***Failed    0.02 sec
[...]
99% tests passed, 2 tests failed out of 356

Total Test time (real) =   4.37 sec

The following tests FAILED:
        131 - export-pg (Failed)
        154 - export-pg-untagged (Failed)
Errors while running CTest