-
-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bump to latest hdf5 version and windows fortran support w/flang #217
base: main
Are you sure you want to change the base?
Conversation
@conda-forge-admin, please rerender |
Hi! This is the friendly automated conda-forge-linting service. I just wanted to let you know that I linted all conda-recipes in your PR ( |
…nda-forge-pinning 2024.04.16.09.40.08
@conda-forge-admin, please rerender |
…nda-forge-pinning 2024.04.16.09.40.08
Hey @h-vetinari, I tried flang-new here in hopes of compiling HDF5 with fortran enabled on windows. Unfortunately it fails during the configuration :( Any ideas where I ought to start looking? (I have posted a question about this on the flang-compiler slack channel also).
Based on
I had hopes that this could work given that it seems LLVM flang compiles HDF5 on linux. |
@conda-forge-admin, please rerender |
…nda-forge-pinning 2024.04.18.09.07.49
Okay, here's another update. Regarding the flang windows error. It seems the previous reported error was a simple CMake config forcing invalid flang comands. I added a patch for that which seems to work. The error we're seeing now is
And by closer inspection of the
it seems this might be related to llvm/llvm-project#77282. |
Hey, here's a small update. Once llvm/llvm-project#89403 is fixed it might be possible to compile HDF5 with fortran enabled on Windows using LLVM FLang. Related to HDFGroup/hdf5#4419 I'll just convert this PR to a draft for now (pending fix for LLVM flang). |
…est-hdf5 # Conflicts: # .ci_support/osx_64_mpimpich.yaml # .ci_support/osx_64_mpinompi.yaml # .ci_support/osx_64_mpiopenmpi.yaml # recipe/bld.bat # recipe/conda_build_config.yaml # recipe/meta.yaml
@conda-forge-admin, please rerender |
…nda-forge-pinning 2024.07.31.04.56.44
Hey @h-vetinari, I am testing the latest flang release candidate (FYI @mjklemm. Thanks for the great work on making flang work with HDF5. I understand you might not be 100% finished with the necessary changes in flang for HDF5 to compile on windows. But I thought I could give it a try nonetheless :) ) Anyways, the new error is You don't suppose this might be related to the conda flang build itself in the rc branch bld.bat? For reference, here is the new error (from the CMakeConfigureLog.yaml in this run): -
kind: "try_run-v1"
backtrace:
- "config/cmake/HDF5UseFortran.cmake:46 (TRY_RUN)"
- "config/cmake/HDF5UseFortran.cmake:149 (FORTRAN_RUN)"
- "CMakeLists.txt:1076 (include)"
directories:
source: "D:/bld/hdf5_1722438888865/work/build/CMakeFiles/CMakeTmp"
binary: "D:/bld/hdf5_1722438888865/work/build/CMakeFiles/CMakeTmp"
cmakeVariables:
CMAKE_EXE_LINKER_FLAGS: "/machine:x64 -stack:10000000"
CMAKE_Fortran_FLAGS: ""
CMAKE_Fortran_FLAGS_RELEASE: ""
CMAKE_MODULE_PATH: "D:/bld/hdf5_1722438888865/work/config/cmake"
CMAKE_POSITION_INDEPENDENT_CODE: "ON"
buildResult:
variable: "COMPILE_RESULT_VAR"
cached: true
stdout: |
Change Dir: 'D:/bld/hdf5_1722438888865/work/build/CMakeFiles/CMakeTmp'
Run Build Command(s): D:/bld/hdf5_1722438888865/_build_env/Library/bin/ninja.exe -v cmTC_3969e
[1/4] C:\\Windows\\system32\\cmd.exe /C "D:\\bld\\hdf5_1722438888865\\_build_env\\Library\\bin\\flang-new.exe -cpp -fms-runtime-lib=dll -E D:\\bld\\hdf5_1722438888865\\work\\build\\CMakeFiles\\CMakeTmp\\testFortranCompiler1.f90 > CMakeFiles\\cmTC_3969e.dir\\testFortranCompiler1.f90-pp.f90 && D:\\bld\\hdf5_1722438888865\\_build_env\\Library\\bin\\cmake.exe -E cmake_ninja_depends --tdi=CMakeFiles\\cmTC_3969e.dir\\FortranDependInfo.json --lang=Fortran --src=CMakeFiles\\cmTC_3969e.dir\\testFortranCompiler1.f90-pp.f90 --out=CMakeFiles\\cmTC_3969e.dir\\testFortranCompiler1.f90-pp.f90 --dep=CMakeFiles\\cmTC_3969e.dir\\testFortranCompiler1.f90-pp.f90.d --obj=CMakeFiles\\cmTC_3969e.dir\\testFortranCompiler1.f90.obj --ddi=CMakeFiles\\cmTC_3969e.dir\\testFortranCompiler1.f90.obj.ddi --src-orig=D:\\bld\\hdf5_1722438888865\\work\\build\\CMakeFiles\\CMakeTmp\\testFortranCompiler1.f90"
[2/4] D:\\bld\\hdf5_1722438888865\\_build_env\\Library\\bin\\cmake.exe -E cmake_ninja_dyndep --tdi=CMakeFiles\\cmTC_3969e.dir\\FortranDependInfo.json --lang=Fortran --dd=CMakeFiles\\cmTC_3969e.dir\\Fortran.dd @CMakeFiles\\cmTC_3969e.dir\\Fortran.dd.rsp
[3/4] D:\\bld\\hdf5_1722438888865\\_build_env\\Library\\bin\\flang-new.exe -ID:\\bld\\hdf5_1722438888865\\work\\build\\CMakeFiles\\CMakeTmp -fms-runtime-lib=dll -ffixed-line-length-72 -o CMakeFiles\\cmTC_3969e.dir\\testFortranCompiler1.f90.obj -c CMakeFiles\\cmTC_3969e.dir\\testFortranCompiler1.f90-pp.f90
[4/4] C:\\Windows\\system32\\cmd.exe /C "cd . && D:\\bld\\hdf5_1722438888865\\_build_env\\Library\\bin\\cmake.exe -E vs_link_exe --intdir=CMakeFiles\\cmTC_3969e.dir --rc=C:\\PROGRA~2\\WI3CF2~1\\10\\bin\\100226~1.0\\x64\\rc.exe --mt="" --manifests -- C:\\PROGRA~1\\MICROS~2\\2022\\ENTERP~1\\VC\\Tools\\MSVC\\1429~1.301\\bin\\HostX64\\x64\\link.exe /nologo CMakeFiles\\cmTC_3969e.dir\\testFortranCompiler1.f90.obj /out:cmTC_3969e.exe /implib:cmTC_3969e.lib /pdb:cmTC_3969e.exe.dbg /version:0.0 /machine:x64 -stack:10000000 /INCREMENTAL:NO /subsystem:console ws2_32.lib wsock32.lib shlwapi.lib -libpath:"D:/bld/hdf5_1722438888865/_build_env/Library/lib" -libpath:"D:/bld/hdf5_1722438888865/_build_env/Library/lib/clang/19/lib/windows" && cd ."
FAILED: cmTC_3969e.exe
C:\\Windows\\system32\\cmd.exe /C "cd . && D:\\bld\\hdf5_1722438888865\\_build_env\\Library\\bin\\cmake.exe -E vs_link_exe --intdir=CMakeFiles\\cmTC_3969e.dir --rc=C:\\PROGRA~2\\WI3CF2~1\\10\\bin\\100226~1.0\\x64\\rc.exe --mt="" --manifests -- C:\\PROGRA~1\\MICROS~2\\2022\\ENTERP~1\\VC\\Tools\\MSVC\\1429~1.301\\bin\\HostX64\\x64\\link.exe /nologo CMakeFiles\\cmTC_3969e.dir\\testFortranCompiler1.f90.obj /out:cmTC_3969e.exe /implib:cmTC_3969e.lib /pdb:cmTC_3969e.exe.dbg /version:0.0 /machine:x64 -stack:10000000 /INCREMENTAL:NO /subsystem:console ws2_32.lib wsock32.lib shlwapi.lib -libpath:"D:/bld/hdf5_1722438888865/_build_env/Library/lib" -libpath:"D:/bld/hdf5_1722438888865/_build_env/Library/lib/clang/19/lib/windows" && cd ."
LINK: command "C:\\PROGRA~1\\MICROS~2\\2022\\ENTERP~1\\VC\\Tools\\MSVC\\1429~1.301\\bin\\HostX64\\x64\\link.exe /nologo CMakeFiles\\cmTC_3969e.dir\\testFortranCompiler1.f90.obj /out:cmTC_3969e.exe /implib:cmTC_3969e.lib /pdb:cmTC_3969e.exe.dbg /version:0.0 /machine:x64 -stack:10000000 /INCREMENTAL:NO /subsystem:console ws2_32.lib wsock32.lib shlwapi.lib -libpath:D:/bld/hdf5_1722438888865/_build_env/Library/lib -libpath:D:/bld/hdf5_1722438888865/_build_env/Library/lib/clang/19/lib/windows /MANIFEST:EMBED,ID=1" failed (exit code 1104) with the following output:
LINK : fatal error LNK1104: cannot open file 'clang_rt.builtins.lib'\x0d
ninja: build stopped: subcommand failed.
exitCode: 1
runResult:
variable: "RUN_RESULT_VAR"
cached: true
And also for reference, here is the previously reported error when compiling with flang v18.1:
Best Regards |
Glad to hear that flang 19 is looking better! Same situation for openblas BTW. :) The way I handle this in scipy currently is as follows: :: need to read clang version for path to compiler-rt
FOR /F "tokens=* USEBACKQ" %%F IN (`clang.exe -dumpversion`) DO (
SET "CLANG_VER=%%F"
)
set "FFLAGS=-D_CRT_SECURE_NO_WARNINGS -D_MT -D_DLL --target=x86_64-pc-windows-msvc"
set "LDFLAGS=--target=x86_64-pc-windows-msvc -fms-runtime-lib=dll -fuse-ld=lld"
set "LDFLAGS=%LDFLAGS% -Wl,-defaultlib:%BUILD_PREFIX%/Library/lib/clang/!CLANG_VER:~0,2!/lib/windows/clang_rt.builtins-x86_64.lib" (though some of that should be possible to remove now, e.g. I've also put this on the list of possible improvements for flang-support in meson. |
Alternatively, you could also try c_compiler: # [win]
- clang # [win]
cxx_compiler: # [win]
- clangxx # [win] where the activation scripts already take care of linking compiler-rt. I've just opened conda-forge/clang-win-activation-feedstock#40 to clean up the default flags there, and I'll follow on flang side later. PS. This will probably conflict for now because the windows compiler activation hasn't been rebuilt for llvm 19 yet; I'll do that after the flags are sorted. In the meantime, you have manual work-arounds as mentioned above. |
With Flang 19.x the error about unresolved external symbols for real_kinds and friends should be gone. When you setup the Flang build, please make sure that you have compiler-rt in LLVM_ENABLE_PROJECTS. Otherwise, Flang somehow tries to pick up the one that comes with Visual Studio and that does not seem to work. |
We cannot build monolithically for several reasons. But the llvm
Yes, mainly because they don't have 128bit support (at least, that was the main reason the last time I looked at this; |
Hey @h-vetinari, I seem to have misunderstood the manual flags you suggested. It seems cmake passes the flags to LINK.exe which doesnt like the flags. I tried changing the flags to MSVC (ie. /DEFAULTLIB etc) but no change. What helped though, was renaming Do you have any suggestions as to what I should do?
|
No you didn't misunderstand, but there was some extra config missing to redirect to LLVM's |
Well, now I'm seriously confused. The same build (
|
OK, I think I got it, I still need to remove The rest of the logs is not visible for some reasons, but the variables do get set. |
Partially reverts "let compiler activation do its job" This reverts commit aad938e.
OK, there seems to be something else involved here:
This fails here, which seems like flang doesn't generate the output that HDF5 expects:
|
Yeah, so the fortran code it tries to run and fails on is this PROGRAM FC08_AVAIL_KINDS
USE, INTRINSIC :: ISO_FORTRAN_ENV, ONLY : stdout=>OUTPUT_UNIT, integer_kinds, real_kinds, logical_kinds
IMPLICIT NONE
INTEGER :: ik, jk, k, max_decimal_prec
INTEGER :: num_rkinds, num_ikinds, num_lkinds
! Find integer KINDs
num_ikinds = SIZE(integer_kinds)
DO k = 1, num_ikinds
WRITE(stdout,'(I0)', ADVANCE='NO') integer_kinds(k)
IF(k.NE.num_ikinds)THEN
WRITE(stdout,'(A)',ADVANCE='NO') ','
ELSE
WRITE(stdout,'()')
ENDIF
ENDDO
! Find real KINDs
num_rkinds = SIZE(real_kinds)
max_decimal_prec = 1
prec: DO ik = 2, 36
exp: DO jk = 1, 700
k = SELECTED_REAL_KIND(ik,jk)
IF(k.LT.0) EXIT exp
max_decimal_prec = ik
ENDDO exp
ENDDO prec
DO k = 1, num_rkinds
WRITE(stdout,'(I0)', ADVANCE='NO') real_kinds(k)
IF(k.NE.num_rkinds)THEN
WRITE(stdout,'(A)',ADVANCE='NO') ','
ELSE
WRITE(stdout,'()')
ENDIF
ENDDO
WRITE(stdout,'(I0)') max_decimal_prec
WRITE(stdout,'(I0)') num_ikinds
WRITE(stdout,'(I0)') num_rkinds
! Find logical KINDs
num_lkinds = SIZE(logical_kinds)
WRITE(stdout,'(I0)') num_lkinds
DO k = 1, num_lkinds
WRITE(stdout,'(I0)', ADVANCE='NO') logical_kinds(k)
IF(k.NE.num_lkinds)THEN
WRITE(stdout,'(A)',ADVANCE='NO') ','
ELSE
WRITE(stdout,'()')
ENDIF
ENDDO
END PROGRAM FC08_AVAIL_KINDS If I try to compile using a simple batch script it fails with the missing :: Make sure you've activated the flang environment
set clanglibdir=%CONDA_PREFIX:\=/%/lib/clang/19/lib/windows
REM It seems whatever I do, I have to rename the clang_rt.builtins-x86_64.lib to clang_rt.builtins.lib
copy "%clanglibdir%\clang_rt.builtins-x86_64.lib" "%clanglibdir%\clang_rt.builtins.lib"
flang-new -L=%CONDA_PREFIX:\=/%/lib/clang/19/lib/windows read_and_integer_kinds.f90 -o read_and_integer_kinds.exe
call read_and_integer_kinds.exe Which outputs
I'll try to investigate further down in the cmake config tomorrow. |
…directory: '/MT'.
It sounds like the |
... which is because the |
Hey @h-vetinari, I was going to make a patch, so I have been trying a few different approaches locally where I force the appropriate flags to the FORTRAN_RUN (and the CMAKE build logs confirm that the flags are passed in). But it still seems like the Are you sure we aren't missing anything else? I tried to simplify this by modifying the previous batch file example I made by doing this: @echo off
setlocal enabledelayedexpansion
:: need to read clang version for path to compiler-rt
FOR /F "tokens=* USEBACKQ" %%F IN (`clang.exe -dumpversion`) DO (
SET "CLANG_VER=%%F"
)
set CL_LIBDIR=%CONDA_PREFIX:/=/%/Library/lib/clang/!CLANG_VER:~0,2!/lib/windows
set "FFLAGS=-D_CRT_SECURE_NO_WARNINGS -D_MT -D_DLL --target=x86_64-pc-windows-msvc"
set "LDFLAGS=-fms-runtime-lib=dll -fuse-ld=lld"
set "LDFLAGS=%LDFLAGS% -Wl,-defaultlib:%CL_LIBDIR%/clang_rt.builtins-x86_64.lib"
set "CLI_ARGS=%FFLAGS% %LDFLAGS%"
flang-new %CLI_ARGS% read_and_integer_kinds.f90 -o read_and_integer_kinds.exe -v
call read_and_integer_kinds.exe
endlocal That ends up with the following error output flang-new version 19.1.0-rc2
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Work\Miniforge3\envs\flangdev\Library\bin
"C:\\Work\\Miniforge3\\envs\\flangdev\\Library\\bin\\flang-new" -fc1 -triple x86_64-pc-windows-msvc19.40.33811 -emit-obj -D _CRT_SECURE_NO_WARNINGS -D _MT -D _DLL -mrelocation-model pic -pic-level 2 -target-cpu x86-64 --dependent-lib=clang_rt.builtins-x86_64.lib -D_MT -D_DLL --dependent-lib=msvcrt --dependent-lib=FortranRuntime.dynamic.lib --dependent-lib=FortranDecimal.dynamic.lib -D_MSC_VER=1940 -D_MSC_FULL_VER=194033811 -D_WIN32 -D_M_X64=100 -resource-dir "C:\\Work\\Miniforge3\\envs\\flangdev\\Library\\lib\\clang\\19" -mframe-pointer=none -o "C:\\Users\\KRISTO~1\\AppData\\Local\\Temp\\read_and_integer_kinds-87ae5c.o" -x f95-cpp-input read_and_integer_kinds.f90
"C:\\Work\\Miniforge3\\envs\\flangdev\\Library\\bin\\lld-link" -out:read_and_integer_kinds.exe "-libpath:C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional\\VC\\Tools\\MSVC\\14.40.33807\\lib\\x64" "-libpath:C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional\\VC\\Tools\\MSVC\\14.40.33807\\atlmfc\\lib\\x64" "-libpath:C:\\Program Files (x86)\\Windows Kits\\10\\Lib\\10.0.26100.0\\ucrt\\x64" "-libpath:C:\\Program Files (x86)\\Windows Kits\\10\\Lib\\10.0.26100.0\\um\\x64" "-libpath:C:\\Work\\Miniforge3\\envs\\flangdev\\Library\\lib" /subsystem:console "-libpath:C:\\Work\\Miniforge3\\envs\\flangdev\\Library\\lib\\clang\\19\\lib\\windows" -nologo "-defaultlib:C:\\Work\\Miniforge3\\envs\\flangdev/Library/lib/clang/19/lib/windows/clang_rt.builtins-x86_64.lib" "C:\\Users\\KRISTO~1\\AppData\\Local\\Temp\\read_and_integer_kinds-87ae5c.o"
lld-link: error: could not open 'clang_rt.builtins.lib': no such file or directory
flang-new: error: linker command failed with exit code 1 (use -v to see invocation)
'read_and_integer_kinds.exe' is not recognized as an internal or external command,
operable program or batch file. Is there something not working with the |
The variables
points to the "wrong" library (compare with LDFLAGS which points to But looking at some of the logs, it appears that lld-link is perhaps in windows-only mode, because otherwise it shouldn't complain about the following arguments:
|
# Conflicts: # .ci_support/win_64_mpiimpi.yaml # .ci_support/win_64_mpinompi.yaml # recipe/meta.yaml
Hi! This is the friendly automated conda-forge-linting service. I just wanted to let you know that I linted all conda-recipes in your PR ( For recipe/meta.yaml:
|
@conda-forge-admin, please rerender |
…nda-forge-pinning 2024.10.12.05.36.28
I thought I might give fortran support for windows a try here as well (seeing as we've had some success in adding fortran support on windows using flang in mumps, mgis, mfront).
However, it failed locally in my initial attempts on windows with
Fortran compiler requires either intrinsic functions SIZEOF or STORAGE_SIZE
If that is not a quick fix, I can take the fortran support out and put it in a different PR.
Checklist
0
(if the version changed)conda-smithy
(Use the phrase@conda-forge-admin, please rerender
in a comment in this PR for automated rerendering)