[HDF5] Cross-compile for all platforms

[giordano]

Don't get excited, this will take a loooong time, if finite at all.

[giordano]

It doesn't even work for x86_64-linux-musl 🤦‍♂️ This will take even longer than what I was initially thinking

[giordano]

@steven-varga I'm sorry for having disappeared for so long, but now I'd like to finally go back to this!

I know you prepared this program that we should run on different platforms to prepare the cross-compilation (thanks for that!), but it's not clear to me how we should use it.

For example, on my machine I get

% make                
DEV_T_IS_SCALAR 1
FC_DUMMY_MAIN 1
GETTIMEOFDAY_GIVES_TZ 1
__GLIBC_PREREQ 1
HAVE_ATTRIBUTE 0
HAVE_C99_DESIGNATED_INITIALIZER 0
HAVE_C99_FUNC 0
HAVE_DEFAULT_SOURCE 0
HAVE_DIRECT 0
HAVE_FUNCTION 0
HAVE_IOEO 1
HAVE_LONG_LONG 1
HAVE_OFF64_T 1
HAVE_SOCKLEN_T 1
HAVE_STAT64_STRUCT 1
HAVE_STAT_ST_BLOCKS 0
HAVE_STRUCT_TEXT_INFO 1
HAVE_STRUCT_TIMEZONE 1
HAVE_STRUCT_VIDEOCONFIG 1
HAVE_SYS_SOCKET_H 1
HAVE_SYS_TIME_H 1
HAVE_SYS_TYPES_H 1
HAVE_TIMEZONE 0
HAVE___TM_GMTOFF 1
HAVE_TM_GMTOFF 0
HAVE_UNISTD_H 1
PRINTF_LL_WIDTH 1
STDC_HEADERS 0
SYSTEM_SCOPE_THREADS 0
TEST_DIRECT_VFD_WORKS 1
TEST_LFS_WORKS 0
TIME_WITH_SYS_TIME 0
VSNPRINTF_WORKS 0
H5_LDOUBLE_TO_LONG_SPECIAL_TEST 1
H5_LONG_TO_LDOUBLE_SPECIAL_TEST 1
H5_LDOUBLE_TO_LLONG_ACCURATE_TEST 0
H5_LLONG_TO_LDOUBLE_CORRECT_TEST 0
H5_NO_ALIGNMENT_RESTRICTIONS_TEST 0
H5_DISABLE_SOME_LDOUBLE_CONV_TEST 1
% make conversion-dump 
FC_DUMMY_MAIN H5_DISABLE_SOME_LDOUBLE_CONV_TEST H5_LDOUBLE_TO_LLONG_ACCURATE_TEST H5_LDOUBLE_TO_LONG_SPECIAL_TEST H5_LLONG_TO_LDOUBLE_CORRECT_TEST H5_LONG_TO_LDOUBLE_SPECIAL_TEST H5_NO_ALIGNMENT_RESTRICTIONS_TEST \n
% make try-dump       
DEV_T_IS_SCALAR FC_DUMMY_MAIN FC_DUMMY_MAIN FC_DUMMY_MAIN GETTIMEOFDAY_GIVES_TZ __GLIBC_PREREQ HAVE_ATTRIBUTE HAVE_C99_DESIGNATED_INITIALIZER HAVE_C99_FUNC HAVE_DEFAULT_SOURCE HAVE_DIRECT HAVE_FUNCTION HAVE_IOEO HAVE_LONG_LONG HAVE_OFF64_T HAVE_SOCKLEN_T HAVE_STAT64_STRUCT HAVE_STAT_ST_BLOCKS HAVE_STRUCT_TEXT_INFO HAVE_STRUCT_TIMEZONE HAVE_STRUCT_VIDEOCONFIG HAVE_SYS_SOCKET_H HAVE_SYS_TIME_H HAVE_SYS_TIME_H HAVE_SYS_TIME_H HAVE_SYS_TIME_H HAVE_SYS_TIME_H HAVE_SYS_TYPES_H HAVE_SYS_TYPES_H HAVE_TIMEZONE HAVE___TM_GMTOFF HAVE_TM_GMTOFF HAVE_UNISTD_H PRINTF_LL_WIDTH STDC_HEADERS SYSTEM_SCOPE_THREADS TEST_DIRECT_VFD_WORKS TEST_LFS_WORKS TIME_WITH_SYS_TIME VSNPRINTF_WORKS \n

which looks also different from what you showed in the README.md for the two *-dump rules.

[steven-varga]

Hi Mose,
it's been a long time! How are you? I downloaded your posted results, then executed diff mose.txt steve.txt, can you find out why I am getting matching result? (as opposed to different one)

I would need the target arch and the result executed on target architecture. Think of a X predictor and Y outcome/target, much similar used in supervised learning; then reverse engineer what is happening.

steve

[giordano]

I ran that command on my machine, x86_64-linux-gnu

[musm]

would it be possible to at least start with cross-compilation just for linux ?

[musm]

@steven-varga We are hoping to upgrade HDF5, but the lack of cross-compilation is proving to be a big burden.

I have had a hard time following exactly what your suggestion is. Do you mind providing some step-by-step instructions so even a dummy like me can follow?

[musm]

@steven-varga In particular, what exactly are we supposed to do with the output from hdf5/config/cmake/ConversionTests.c (from https://github.com/steven-varga/HDFGroup-mailinglist/tree/527f2245d3ac1d13e2fd75810955938c3aeb8599/crosscompile-2020-mar-25#hdf5configcmakeconversiontestsc)
?

[musm]

I have the output from the requested program for 3 platforms in my repo: https://github.com/musm/hdf5-build-output

[steven-varga]

Thank you for contributing! The idea is to build a database of the environment variables, then to examine if there is any difference among them and use them to initialise CMAKE (possibly GNU configure as well).

In the current HDF5 library compile setting -- where cross compiling doesn't work -- these values are obtained by executing the program on the target platform. This mechanism doesn't work on a pure cross compile environment, unless you provide a platform + cpu simulator as well. However by pre-executing the test scripts -- as you just did -- we already have them environment variables, no execution on given target needed, instead we can use the database.

The current state already can get us a working solution, but more the better -- as I can reverse what is the original idea, and if all the records are matching possibly reason that this functionality is outdated, and should be replaced with a simpler mechanism.

Please give me few weeks to actively look at this problem, as currently my hands are full with two vaguely related projects.

best:steve

[musm]

@steven-varga That makes sense to me. Thank you for describing the general idea, which makes sense to me now.

Yes, I think I can manage to get the output for all the required platforms in the meantime. We already have the output from 4 target platforms in the linked repo. I'm looking forward to finally being able to handle this.

[musm]

@steven-varga One point of clarification. Are the generated variables from you're test program environment variables or what? In particular, how do you plan to use them?

[steven-varga]

One point of clarification. Are the generated variables from you're test program environment variables or what? In particular, how do you plan to use them?

carefully :) -- if you set these variables to the desired values, then disable the compile + execute phase of the build process test programs, then cross compiling will work for that given platform -- by virtue.

This statement is based on my research, brief email exchanges with Allen Byrn and online meetings with Gerd Heber, according to Allen (THDFGroup, cmake specialist) the CMAKE version is the easier to accomplish (as opposed to GNU configure). -- and yes, we are aware of an attempt with GNU configure.

More variety of platform is the above scripts is executed, the stronger the reasoning is that the current approach is outdated, and an official alternative solution could be simpler/better. Hence I am reaching out to the community to get the platform specific results of these tests scripts, originally lifted from the HDF5 code base.

steve,
h5cpp.org

[musm]

Thank you for the clarification. We managed to obtain many more platforms over at https://github.com/musm/hdf5-build-output/tree/master/outputs hopefully these prove to be helpful.

[steven-varga]

Hi @musm I looked at the at your repository and noticed my work from this HDFGroup mailing list re-posted without references.

Can you please correct it.
steve

[musm]

@steven-varga is this ok with you https://github.com/musm/hdf5-build-output/blob/master/README.md ? (note the copyright notices are available in each individual file as well). Let me know if I need to add anything else for copyright.

[musm]

Studying the output from all the posted platforms.

Here are the differences (all other variables are the same among the platforms in the repo)

HAVE_DEFAULT_SOURCE 0  (1 on x86_64-apple-darwin19.6.0)
HAVE_IOEO 1 (0 on i686-pc-cygwin and x86_64-pc-cygwin)
HAVE_STAT64_STRUCT 1 (0 on x86_64-apple-darwin19.6.0)
HAVE___TM_GMTOFF 1 (0 on i686-linux-musl and x86_64-pc-cygwin)
TEST_DIRECT_VFD_WORKS 1 (0 on i686-linux-musl and i686-pc-cygwin and x86_64-linux-musl)

[simonbyrne]

What's the current status of this? There is a new release of HDF5 (1.12.1), there are a lot of build system changes which may or may not help:
https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.1/src/hdf5-1.12.1-RELEASE.txt

[ViralBShah]

Is this worth trying again with a recent version?

[cbldlpl]

any update for it?

[giordano]

Is this worth trying again with a recent version?

Tried, but zero progress, still plenty of TRY_RUNs.

[steven-varga]

Somehow I lost track of this problem: @musm thank you for the output and the analysis.
The bottom line: for me it is hard to judge if this is important or not, you see it is quite a work in terms of expertise to make it happen:

• one has to do the modifications
• convince HDF5 people to accept the modification
• convince Julia people to use it

This issue has been closed server times, which confused me a bit: is this important or not so much. Can someone authoritative state here: yes/no, please? (I honestly do not know the significance of this, I just know how to do it)

[giordano]

@steven-varga HDFGroup/hdf5#1203 (comment) made me think there was some progress upstream, but I don't see it. I also couldn't find the promised documentation.

[ViralBShah]

@steven-varga This is certainly an important PR. The reason for closing and opening multiple times is because we are unsure of whether it will wok or not, and sometimes to trigger PRs. It may be better for you to make a new PR if you have a way to do it, which may be easier for you to work with.

[musm]

@steven-varga
There will be no convincing for Julia people to use it! We will use it, since it will be a big benefit instead of the present monkey-patch solution.

[steven-varga]

OK: I am travelling away for two weeks, once I am back, I will consider this case re-opened, and make this happen.

[simonbyrne]

So based on the latest comments on HDFGroup/hdf5#1203 (comment), do we have a path forward here?

[giordano]

That...doesn't sound like a good solution at all.

[giordano]

Closing in favour of #6551