Compile julia with icc and ifort and use MKL

[ViralBShah]

Add support to compile julia with icc and ifort using USEICC and USEIFC.

[Keno]

This isn't right, the ifort program does not have the prefix.

[tkelman]

Can ifort cross compile at all?

[Keno]

Yes, there's a -m<stuff> flag for various things.

[ViralBShah]

arpack tests work now.

[tkelman]

In your testing, are you using MKL and Intel's libm too? Or some other combination of OpenBLAS and/or openlibm?

[ViralBShah]

I am only using Intel compilers and MKL for now. Not yet using the intel libm. I guess we should just enable intel libm too, but that will have the rem_pio2 issue.

[ViralBShah]

Perhaps we should have one Makefile variable for using all intel compilers and math libraries.

[StefanKarpinski]

That would be nice – you should also be able to choose them a la carte, but having a meta variable that can flip all the other default would be handy.

[tkelman]

Thanks, curious to get a handle on what is working and what isn't. I agree with Stefan.

So most of what you needed to avoid the problems @morrisonlevi saw in #3902 (comment) was a few extra CFLAGS for openlibm to behave properly with icc?

[ViralBShah]

Yes, with optimizations turned on, icc truncates denormals to zero which fails our tests. I have flags to avoid that now. Also, the correct fortran flags. I don't see the failures that @morrisonlevi sees in #3902. I suspect because my flags turn on strict adherence to IEEE754, which openlibm assumes.

The build status here is of no use (we probably should have an icc build environment on travis too), but all tests pass for me in this PR.

Cc @staticfloat

[ViralBShah]

I can't use MKL's FFTW wrappers, because we use fftw_alignment_of, which MKL doesn't seem to have.

[ViralBShah]

Of course, we should have one flag to use all Intel libraries, and also the a la carte options.

[StefanKarpinski]

Maybe @stevengj can port that to Julia (or someone else can) so that we don't depend on FFTW's.

[ViralBShah]

There may be other FFTW functions we call that MKL may not have wrappers for. @stevengj perhaps knows if it is realistic for us to be able to use MKL's FFTW wrappers.

[pao]

Yes, with optimizations turned on, icc truncates denormals to zero which fails our tests.

Depending on whether it's set when the library is loaded or when the function is called, you could try ccall(:jl_zero_subnormals, Uint8, (Uint8,), 0) before running tests?

[ViralBShah]

I think we should probably expose that in Base and document it. It is generally useful.

[ViralBShah]

MKL just seems to be missing the fftw_alignment_of and fftw_set_timelimit functions.

[StefanKarpinski]

fftw_alignment_of just takes the address mod 16 and we can probably just ignore fftw_set_timelimit.

[ViralBShah]

Actually, fftw_set_timelimit does exist, and fftw_alignment_of is not part of the documented FFTW API. This should work out.

[pao]

I think we should probably expose that in Base and document it.

#3255. But it should probably work like @inbounds.

[ViralBShah]

Yes - good point. Let's discuss that in the relevant issue.

[stevengj]

@StefanKarpinski, technically fftw_alignment_of takes the address mod 16 only if SIMD is enabled, but it is safe to assume that SIMD is enabled if you are using MKL.

[ViralBShah]

I saw that in the fftw codebase yesterday. Should I have a Julia implementation that does that when mkl is being used, or just make it a noop when using mkl?

[stevengj]

The MKL FFT documentation is not very clear on this point, but I get the impression that their "plans" may work on arbitrary input arrays regardless of alignment (albeit possibly suboptimally). In this case it would be safe to define the alignment checks to a no-op when using MKL.

(But it probably wouldn't hurt to define them to enforce 16-byte alignment. As I said, if you are using MKL then you are almost certainly using SIMD.)

[stevengj]

It looks like MKL used to require 16-byte alignment, but eliminated this requirement in version 10.2 update 3 (released in 2009).

[ViralBShah]

@andreasnoackjensen @tkelman @morrisonlevi Can you guys try out this branch? I now have all MKL stuff working. My Make.user has this:

override USEICC=1
override USEIFC=1
override USE_INTEL_MKL=1
override USE_INTEL_MKL_FFT=1
override USE_INTEL_LIBM=1

[tkelman]

I have MKL but not the Intel compilers, I was able to get things to build and link with MKL and GNU compilers, but arpack segfaults in mkl_blas_zdotc_htn. I could try fiddling with the mkl_gf libraries but it might be more trouble than it's worth for now. If all-Intel works, that's a big step.

Wonder what the chances are of convincing Intel to let Travis provide a VM image with icc/ifort/mkl all set up...

[ViralBShah]

We do have licences, which we never got around to using before.

[ViralBShah]

@lindahua Could you try this out?

[morrisonlevi]

I am downloading the dependencies now; will report back on that soon.

Also, it would be nice if we could compile many of the libraries in statically if available; the reason is that at HPC sites you have many compilers and libraries available and if you want to use Julia and software X in the same job where X uses different versions of libraries it gets hairy to manage. So at least -static-intel might be helpful. Just something to think about.

[morrisonlevi]

It still tried to use GCC by the way. I had to use CC/FC/CXX variables. Maybe that was implied, but I figured that's what the USEICC meant.

[StefanKarpinski]

We should be able to reduce fully static binaries too pretty soon.

[ViralBShah]

-static-intel is a good idea. Will add that. You have to use both, USEICC and USEIFC. I will take a look again.

[ViralBShah]

I am going to merge this for now and explore -static-intel separately.

[ViralBShah]

The build failures are not relevant.

[tkelman]

This is relevant though - on Windows, the rem_pio2 functions are now the only functions exported from libopenspecfun.dll:

exception on 2: ERROR: ccall: could not find function Faddeeva_erfcx_re in library libopenspecfun
while loading math.jl, in expression starting on line 39
ERROR: ccall: could not find function Faddeeva_erfcx_re in library libopenspecfun
while loading math.jl, in expression starting on line 39

[tkelman]

It seems if some functions in the library have DLLEXPORT then only they get exported, but if none of the functions have it, then they all get exported? That fixes it on my end, I'm going to remove the DLLEXPORTs in rem_pio2 then bump the submodule.

[ViralBShah]

Ok, please do that. This is good to know for the future.

[ViralBShah]

Perhaps we should tag the releases of openblas and openspecfun too with the new versions, once all this clears up.