GMT #5199

boriskaus · 2022-07-21T16:57:03Z

This creates a BinaryBuilder version of GMT, the Generic Mapping Tools.

…t on windows as the dependency NetCDF is not available on windows yet

G/GMT/build_tarballs.jl

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

boriskaus · 2022-07-23T21:38:12Z

The errors on windows are:

[21:27:19] CMakeFiles/gmtlib.dir/objects.a(gmt_fft.c.obj):gmt_fft.c:(.text+0x3825): undefined reference to `__imp_gethostname'
[21:27:19] CMakeFiles/gmtlib.dir/objects.a(gmt_init.c.obj):gmt_init.c:(.text+0x3d248): undefined reference to `__imp__set_fmode'
[21:27:19] collect2: error: ld returned 1 exit status
[21:27:19] make[2]: *** [src/CMakeFiles/gmtlib.dir/build.make:2561: src/gmt.dll] Error 1

if that rings a bell for someone, suggestions are helpful!

giordano · 2022-07-23T21:45:37Z

For gethostname I guess you need to link to -lws2_32? Not sure about _set_fmode, documentation doesn't indicate what you need to link to 😕

giordano · 2022-07-23T21:51:40Z

According to

julia --compile=min -e 'using BinaryBuilderBase; BinaryBuilderBase.runshell(Platform("x86_64", "windows"))'
sandbox:${WORKSPACE} # for file in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/*.a; do nm $file | grep __imp__set_fmode && echo "found in ${file}"; done
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libmsvcr100.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libmsvcr110.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libmsvcr120.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libmsvcr120_app.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libmsvcr120d.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libmsvcr90.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libmsvcr90d.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libucrt.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libucrtbase.a

_set_fmode is provided by a bunch of internal libraries, I'm not sure you're supposed to manually link to any of them.

boriskaus · 2022-07-24T09:17:06Z

thanks. If I manually compile this with

-lws2_32 -lmsvcr100

it works & the whole code compiles. Do you happen to know how I can tell CMake to take this libraries into account as well?

giordano · 2022-07-24T10:47:12Z

You have to add the libraries names to the target which needs them, for example with target_link_libraries or whatever they're using to specify the dependencies. ws2_32 is definitely public API, but msvcr100 is something very internal (like all other libraries which provide _set_fmode), you just picked up a random library from the list I shared, that doesn't sound right. I have no idea how _set_fmode is supposed to be properly linked.

G/GMT/build_tarballs.jl

G/GMT/bundled/patches/win_mingw.patch

giordano · 2022-07-24T11:09:14Z

Some creative uses of linker flags:

[21:27:01] /opt/bin/aarch64-linux-gnu-libgfortran4-cxx11/aarch64-linux-gnu-gcc [...] -Wl,-rpath,/usr/local/lib::::::::::::::::::::::: [...]
[...]
[21:27:04] /opt/bin/aarch64-linux-gnu-libgfortran4-cxx11/aarch64-linux-gnu-gcc [...] -Wl,-rpath,/usr/local/lib:/workspace/srcdir/gmt-6.2.0/build/src: [...]

Linking a source directory?!?!?

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

G/GMT/build_tarballs.jl

boriskaus · 2022-07-24T17:24:54Z

ok, I keep being confused with CMake. If I manually add this at the end:

/opt/bin/x86_64-w64-mingw32-libgfortran4-cxx11/x86_64-w64-mingw32-gcc --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -std=gnu99  -shared -o gmt.dll -Wl,--out-implib,gmt.dll.a -Wl,--major-image-version,6,--minor-image-version,2 -Wl,--whole-archive CMakeFiles/gmtlib.dir/objects.a -Wl,--no-whole-archive @CMakeFiles/gmtlib.dir/linklibs.rsp -lws2_32 -lmsvcr100

it works.
Yet, any attempt to add the libraries as flags results in something like

/opt/bin/x86_64-w64-mingw32-libgfortran4-cxx11/x86_64-w64-mingw32-gcc --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -std=gnu99  -lws2_32 -lmsvcr100 -shared -o gmt.dll -Wl,--out-implib,gmt.dll.a -Wl,--major-image-version,6,--minor-image-version,2 -Wl,--whole-archive CMakeFiles/gmtlib.dir/objects.a -Wl,--no-whole-archive @CMakeFiles/gmtlib.dir/linklibs.rsp

which does not work.

joa-quim · 2023-03-02T14:56:24Z

Could you give it another try? I fixed the gethostname and the _set_fmode issues upstream where I was finally able to build GMT with MinGW. But note that those changes are in GMT#master only

giordano · 2023-03-02T15:08:45Z

There are still lots of mistakes in the source code of GMT: all occurrences of WIN32 should be either _WIN32 if they apply to any Windows target, or _MSC_VER if it's only for the Microsoft Visual Studio Code compiler. You shouldn't even expect compilers to wrongly define macros not prefixed with a leading underscore, that's forbidden by the C standard, but non-standard compliant compilers like MSVC might do what they want. Besides, WIN32 isn't even documented in https://learn.microsoft.com/en-us/cpp/preprocessor/predefined-macros?view=msvc-170

joa-quim · 2023-03-02T17:06:32Z

Hmm, let me see if I understand it. You revising(???) the GMT C code and not accepting it and therefore this attempt of creating a GMT_JLL cannot proceed while you do not approve GMT C, right?

We may all be very dumb on GMT (I'm not by far the largest contributor) but we are are a bit aware of the WIN32 vs _WIN32 mess.
https://github.com/GenericMappingTools/gmt/blob/master/src/gmt_notposix.h#L148

And you may also do not like (or believe) to hear that along the GMT developing time we have found dozens of memory access bugs that were detected only by the non-standard compliant compilers like MSVC and that gcc and clang had happily let go.

giordano · 2023-03-02T17:20:33Z

No, you do not understand correctly. I am pointing out factual errors in the source code which prevent correct compilation. We cannot merge a PR if compilation fails for a platform (the alternative being excluding that platform altogether). And compilation for Windows was failing here 🤷

joa-quim · 2023-03-02T17:25:54Z

And compilation for Windows was failing here

Yes, I'm aware of that but the point is: Is it still failing after the upstream fixes? And if yes, where/why.

giordano · 2023-03-02T17:27:06Z

No, I haven't tried, but I just looked for obvious mistakes.

joa-quim · 2023-03-02T17:28:33Z

but I just looked for obvious mistakes.

We could continue discussing this for ever, which I will not. At least while the tone maintains.

giordano · 2023-03-07T23:57:13Z

In case it wasn't obvious, libblastrampoline-5.dll is not the same as libblastrampoline-5-0-2.dll

joa-quim · 2023-03-08T00:03:00Z

Yes, I know. And on Windows the dlls have embedded their own names and that' why we cannot simple rename them and expect it to work.

giordano · 2023-03-08T00:04:28Z

No one is expecting anything to magically work, that has to be fixed, no one has done it.

joa-quim · 2023-03-08T00:18:23Z

Sure. Not blaming anyone.

joa-quim · 2023-03-08T14:30:26Z

I found the libblastrampoline-5-0-2.dll file here, downloaded and tried (again with some quick patchy solution) but got an error saying it's not able to load the gmt.dll due to missing symbol (don't have the right error message anymore). Then I tried the DependencyWalker and found things like this

Does that red on libhdf5_hd-0.dll means it might be a cause for the gmt.dll failure? There is another red in libsz.dll

boriskaus · 2023-03-10T06:31:58Z

To get back to this discussion, it seems to me that at least the gmt binary works just fine on windows as shown below.

Version;

julia> versioninfo()
Julia Version 1.8.3
Commit 0434deb161 (2022-11-14 20:14 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × 12th Gen Intel(R) Core(TM) i3-1215U
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, goldmont)
  Threads: 1 on 8 virtual cores

Installing the GMT_jll from @giordano

PS C:\Users\boris\.julia\dev\GMT_jll> julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.3 (2022-11-14)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.8) pkg> dev https://github.com/giordano/GMT_jll.jl

Running it;

PS C:\Users\boris\.julia\dev\GMT_jll> julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.3 (2022-11-14)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.8) pkg> activate .
  Activating project at `C:\Users\boris\.julia\dev\GMT_jll`

julia> using GMT_jll

julia> run(GMT_jll.gmt())

        GMT - The Generic Mapping Tools, Version 6.5.0_19413d4_2023.03.03 [64-bit] [MP] [8 cores]
        (c) 1991-2023 The GMT Team (https://www.generic-mapping-tools.org/team.html).

        Supported in part by the US National Science Foundation (http://www.nsf.gov/)
        and volunteers from around the world.

        GMT is distributed under the GNU LGPL License (http://www.gnu.org/licenses/lgpl.html).
        Dependencies: netCDF, GDAL, PCRE, FFTW, LAPACK, ZLIB, Ghostscript, GraphicsMagick, FFmpeg.

usage: gmt [options]
       gmt <module name> [<module-options>]

options:
  --help              List descriptions of available GMT modules.
  --new-script[=L]    Write GMT modern mode script template to standard output.
                      Optionally specify bash|csh|batch [Default is current shell].
  --new-glue=name     Write C code for external supplements to glue them to GMT.
  --show-bindir       Show directory with GMT executables.
  --show-citation     Show the most recent citation for GMT.
  --show-classic      Show all classic module names.
  --show-classic-core Show all classic module names (core only).
  --show-cores        Show number of available cores.
  --show-datadir      Show directory/ies with user data.
  --show-dataserver   Show URL of the remote GMT data server.
  --show-dcw          Show the DCW data version used.
  --show-doi          Show the DOI for the current release.
  --show-gshhg        Show the GSHHG data version used.
  --show-library      Show path of the shared GMT library.
  --show-modules      Show all modern module names.
  --show-modules-core Show all modern module names (core only).
  --show-plugindir    Show directory for plug-ins.
  --show-sharedir     Show directory for shared GMT resources.
  --show-userdir      Show full path of user's ~/.gmt dir
  --version           Print GMT version number.

if <module-options> is '=' we call exit (0) if module exist and non-zero otherwise.

Process(setenv(`'C:\Users\boris\.julia\artifacts\b00f4bcc0d779dfd70def088d9a629941a6e23b1\bin\gmt.exe'`,["PATH=C:\\Users\\boris\\AppData\\Local\\Programs\\Julia-1.8.3\\bin;C:\\Users\\boris\\.julia\\artifacts\\0235fabbecfa4e38a3c8e87491b9d4f3e7fe9848\\bin;C:\\Users\\boris\\.julia\\artifacts\\701aaf00cce776fdefff2d993e09800fab61e884\\bin;C:\\Users\\boris\\.julia\\artifacts\\a5b7c195243758d88c64cd8547e527fbace3877b\\bin;C:\\Users\\boris\\.julia\\artifacts\\431cad9bf95e817b5fc813666e353ae36b2202c0\\bin;C:\\Users\\boris\\.julia\\artifacts\\4543a35904e5905f4c387199cc080e27cc202cec\\bin;C:\\Users\\boris\\.julia\\artifacts\\c024ee55cc5bbe84dd4411a68b02dd9a3797fdcd\\bin;C:\\Users\\boris\\.julia\\artifacts\\4ad4b701a165bfb1dec449667cffd85d0a16dcc2\\bin;C:\\Users\\boris\\.julia\\artifacts\\ac8a1c48a02b397f6ce1beb6473de8bbb3b1e0ce\\bin;C:\\Users\\boris\\.julia\\artifacts\\2e9a29903fddf79371cd5f82327f70ada670a234\\bin;C:\\Users\\boris\\.julia\\artifacts\\17025344af7bdfe8a7ac4e8887f0953f1dc50376\\bin;C:\\Users\\boris\\.julia\\artifacts\\01b969d96551292e78f0acfa204f120b09f4e7b4\\bin;C:\\Users\\boris\\.julia\\artifacts\\f2d9050c5dba019ebf272fe7da44f78c5491672d\\bin;C:\\Users\\boris\\.julia\\artifacts\\bcdd265e21bd7089238f661440be60aed9e9e696\\bin;C:\\Users\\boris\\.julia\\artifacts\\511fc12b661678285fd116ffcee88cbaf043b869\\bin;C:\\Users\\boris\\.julia\\artifacts\\72037f1415327f12277eed0521c861517630a0ba\\bin;C:\\Users\\boris\\.julia\\artifacts\\a845328b7acbc4442b321885d186869a90297089\\bin;C:\\Users\\boris\\.julia\\artifacts\\41eb25ea50f277f03154d52354602687fb1abc84\\bin;C:\\Users\\boris\\.julia\\artifacts\\18a38c3afb63ea0df1832d3f946b21b58ba401ea\\bin;C:\\Users\\boris\\.julia\\artifacts\\5d74056071181523cbb62efeddd86132d4db12de\\bin;C:\\Users\\boris\\.julia\\artifacts\\db3d60d39813bf806d1b4a8032e3a2f892c75738\\bin;C:\\Users\\boris\\.julia\\artifacts\\b62e4d49f0fa9e8d9daccda7af8ed7315580d6dc\\bin;C:\\Users\\boris\\.julia\\artifacts\\9c9d5d21081d79fb66bb9aaeea87960a22c5599b\\bin;C:\\Users\\boris\\.julia\\artifacts\\3628616b5a5a8fd8628079ae4fb0d8948fdb37bc\\bin;C:\\Users\\boris\\.julia\\artifacts\\a214c5171a32bff0583c70a08027d44e8ef22cc2\\bin;C:\\Users\\boris\\.julia\\artifacts\\dcd0721a2a49cfb8916395419701035173b57955\\bin;C:\\Users\\boris\\.julia\\artifacts\\83d62d00715073512a2e76bbb95e027bf701a4e3\\bin;C:\\Users\\boris\\.julia\\artifacts\\b7dd1809d0626eac3bf6f97ba8ccfbb6cc63c509\\bin;C:\\Users\\boris\\.julia\\artifacts\\5e6cdd29fca31b08026313af46c1db0c724ebbb2\\bin;C:\\Users\\boris\\.julia\\artifacts\\f572524225d8a9266c31227f4e28a0a1091bd882\\bin;C:\\Users\\boris\\.julia\\artifacts\\a97c68e84f8d803b96763aa872ed49381fb9cb7e\\bin;C:\\Users\\boris\\.julia\\artifacts\\5af028f5ee6fc5d6923b4769eca4bc199914e6bd\\bin;C:\\Users\\boris\\.julia\\artifacts\\38d4f07d7bebeb1d35473280d1ef56f7d8e6991d\\bin;C:\\Users\\boris\\.julia\\artifacts\\e5872943f1078a6397a4f4d0f760db959ec50630\\bin;C:\\Users\\boris\\.julia\\artifacts\\f9f799e6ae3fe7f244bb55052db7ff1cea876c27\\bin;C:\\Users\\boris\\.julia\\artifacts\\4f67e82ee394e74ca63b7dc589dcc9218c12b20a\\bin;C:\\Users\\boris\\.julia\\artifacts\\6a5456c156251deff41ae40ac8dfbb3d8c808ffe\\bin;C:\\Users\\boris\\.julia\\artifacts\\c916a560fa109a4db3d6938a8ab1a6a8b4a09731\\bin;C:\\Users\\boris\\.julia\\artifacts\\3aff78028a0bb2ad40ebc3ae7e465a52dcbe2730\\bin;C:\\Users\\boris\\.julia\\artifacts\\ee9d999d28081333cf8ec10830fd77b75359226b\\bin;C:\\Users\\boris\\.julia\\artifacts\\2ae55b726ee1f4063216b19a1ffb59d3d5b0a6eb\\bin;C:\\Users\\boris\\.julia\\artifacts\\017f69ccc5ddfc0405e5bf8326a572300498e06c\\bin;C:\\Users\\boris\\.julia\\artifacts\\1b573c0aaf4a3a73f296a61c2cacbee17ac0d4e3\\bin;C:\\Users\\boris\\.julia\\artifacts\\7d57f5f863fd09f31bbff6d08d53bfc07f49c3e6\\bin;C:\\Users\\boris\\.julia\\artifacts\\6f5e7c4e3249d2073c64e6beaa3abcd7ae4fa9bc\\bin;C:\\Users\\boris\\.julia\\artifacts\\0c393ed4faa8cdf04f992ff605abaed1c1172f69\\bin;C:\\Users\\boris\\.julia\\artifacts\\160ce27d1af09edffbb7d9f1833f6d3bdefad491\\bin;C:\\Users\\boris\\.julia\\artifacts\\c6e48d3563556547ad4cc0021c2cffa066a079df\\bin;C:\\Users\\boris\\.julia\\artifacts\\da2e0ac84abbe712490f8ee73aca3d512f1eaaab\\bin;C:\\Users\\boris\\.julia\\artifacts\\5b9eb98a95e718b9ddc16d01fd70b19736b13e0b\\bin;C:\\Users\\boris\\.julia\\artifacts\\e2bc28a38e71d07d41a27f8d4b3132c5d1b5a22d\\bin;C:\\Users\\boris\\.julia\\artifacts\\cca2b0693e0ae8df2daca37d0779bb7209f32c5f\\bin;C:\\Users\\boris\\.julia\\artifacts\\b00f4bcc0d779dfd70def088d9a629941a6e23b1\\bin;C:\\Users\\boris\\AppData\\Local\\Programs\\Julia-1.8.3\\bin\\..\\lib\\julia;C:\\Users\\boris\\AppData\\Local\\Programs\\Julia-1.8.3\\bin\\..\\lib;C:\\Users\\boris\\AppData\\Local\\Programs\\Julia-1.8.3\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Program Files\\dotnet\\;C:\\Program Files\\Git\\cmd;C:\\Users\\boris\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\boris\\AppData\\Local\\Programs\\Microsoft VS Code\\bin;C:\\Users\\boris\\AppData\\Local\\Programs\\Julia-1.8.3\\bin;C:\\ProgramData\\boris\\GitHubDesktop\\bin", "USERDOMAIN_ROAMINGPROFILE=DESKTOP-488U587", "HOMEPATH=\\Users\\boris", "VSCODE_INJECTION=1", "PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL", "SESSIONNAME=Console", "SYSTEMROOT=C:\\WINDOWS", "APPDATA=C:\\Users\\boris\\AppData\\Roaming", "PSMODULEPATH=C:\\Users\\boris\\Documents\\WindowsPowerShell\\Modules;C:\\Program Files\\WindowsPowerShell\\Modules;C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules", "COMMONPROGRAMW6432=C:\\Program Files\\Common Files"  …  "PROGRAMFILES(X86)=C:\\Program Files (x86)", "VSCODE_GIT_ASKPASS_MAIN=c:\\Users\\boris\\AppData\\Local\\Programs\\Microsoft VS Code\\resources\\app\\extensions\\git\\dist\\askpass-main.js", "PROGRAMFILES=C:\\Program Files", "CHROME_CRASHPAD_PIPE_NAME=\\\\.\\pipe\\LOCAL\\crashpad_30280_YPJHJPUITZAYFZRU", "LOGONSERVER=\\\\DESKTOP-488U587", "DRIVERDATA=C:\\Windows\\System32\\Drivers\\DriverData", "ORIGINAL_XDG_CURRENT_DESKTOP=undefined", "FONTCONFIG_PATH=C:\\Users\\boris\\.julia\\artifacts\\c916a560fa109a4db3d6938a8ab1a6a8b4a09731\\etc\\fonts", "SYSTEMDRIVE=C:", "PROCESSOR_ARCHITECTURE=AMD64"]), ProcessExited(0))

So I think this can be merged.

joa-quim · 2023-03-10T16:05:05Z

Yes, but the problem is that the gmt.dll fails to load like I reported above. And also in (messy) WKL I have this

julia> using LibCURL_jll, NetCDF_jll, GDAL_jll, FFTW_jll, PCRE_jll, LAPACK32_jll, OpenBLAS32_jll, FFMPEG_jll, Glib_jll, Ghostscript_jll, CompilerSupportLibraries_jll, LLVMOpenMP_jll, GMT_jll
ERROR: InitError: could not load library "/home/j/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so"
/lib/x86_64-linux-gnu/libcurl.so.4: version `CURL_4' not found (required by /home/j/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so)

I wonder if it has something to do with JuliaLang/julia#48419

Note, I was able to produce a GMT_jll for linux with Codespaces (but not with WSL) and that's what I was trying to test.

boriskaus · 2023-03-10T18:32:14Z

On my machine the above command works fine:

julia> using LibCURL_jll, NetCDF_jll, GDAL_jll, FFTW_jll, PCRE_jll, LAPACK32_jll, OpenBLAS32_jll, FFMPEG_jll, Glib_jll,  Ghostscript_jll, CompilerSupportLibraries_jll, LLVMOpenMP_jll, GMT_jll
julia>

perhaps there is something else on your machine that interferes?

FWIW;

julia> NetCDF_jll.libnetcdf_path
"C:\\Users\\boris\\.julia\\artifacts\\4543a35904e5905f4c387199cc080e27cc202cec\\bin\\libnetcdf-19.dll"

julia> NetCDF_jll.LIBPATH_list
6-element Vector{String}:
 "C:\\Users\\boris\\AppData\\Local\\Programs\\Julia-1.8.3\\bin"
 "C:\\Users\\boris\\.julia\\artifacts\\0235fabbecfa4e38a3c8e87491b9d4f3e7fe9848\\bin"
 "C:\\Users\\boris\\.julia\\artifacts\\701aaf00cce776fdefff2d993e09800fab61e884\\bin"
 "C:\\Users\\boris\\.julia\\artifacts\\a5b7c195243758d88c64cd8547e527fbace3877b\\bin"
 "C:\\Users\\boris\\.julia\\artifacts\\431cad9bf95e817b5fc813666e353ae36b2202c0\\bin"
 "C:\\Users\\boris\\.julia\\artifacts\\4543a35904e5905f4c387199cc080e27cc202cec\\bin"

julia> using LAPACK32_jll
julia> LAPACK32_jll.liblapack_path
"C:\\Users\\boris\\.julia\\artifacts\\f572524225d8a9266c31227f4e28a0a1091bd882\\bin\\liblapack.dll"
julia>

works on julia 1.9.0-rc1 as well

joa-quim · 2023-03-10T22:57:49Z

And was it with the libblastrampoline-5-0-2.dll downloaded from the site I linked above or without it?

Anyway, I just noticed that we need to add the PROJ_jll dependency too (I access to its shared library directly).
And do you know how to run the gmt executable with options? If it worked would be run(GMT_jll.gmt("--user-dir")). I need to capture the output of a couple of those options.

boriskaus · 2023-03-11T10:43:01Z

No I am not using any additionally installed packages, but rather the ones that were used to compile GMT_jll, to ensure that all the packages are compatible.

I´m using windows 11 for this along with the GMT_jll build that was helpfully provided by @giordano
Here the steps to reproduce what I did:

Download a development version of GMT

 julia> ]dev https://github.com/giordano/GMT_jll.jl
   Cloning git-repo `https://github.com/giordano/GMT_jll.jl`
   Resolving package versions...
  No Changes to `C:\Users\boris\.julia\environments\v1.8\Project.toml`
  No Changes to `C:\Users\boris\.julia\environments\v1.8\Manifest.toml`

Change to the directory

 julia> ; 
shell> cd .julia\\dev\\GMT_jll\\
C:\Users\boris\.julia\dev\GMT_jll

Activate project

 julia> ]
(@v1.8) pkg> activate .
  Activating project at `C:\Users\boris\.julia\dev\GMT_jll`

Precompile

(GMT_jll) pkg> precompile
    Updating registry at `C:\Users\boris\.julia\registries\General.toml`
    Updating `C:\Users\boris\.julia\dev\GMT_jll\Project.toml`
  [692b3bcd] + JLLWrappers v1.4.1
  [b22a6f82] + FFMPEG_jll v4.4.2+2
  [f5851436] + FFTW_jll v3.3.10+0
  [a7073274] + GDAL_jll v301.600.200+0
  [61579ee1] + Ghostscript_jll v9.55.0+4
  [7746bdde] + Glib_jll v2.74.0+2
  [17f450c3] + LAPACK32_jll v3.10.0+0
  [1d63c593] + LLVMOpenMP_jll v15.0.4+0
  [7243133f] + NetCDF_jll v400.902.5+1
⌅ [656ef2d0] + OpenBLAS32_jll v0.3.17+0
  [2f80f16e] + PCRE_jll v8.44.0+0
  [e66e0078] ~ CompilerSupportLibraries_jll ⇒ v0.5.2+0
  [deac9b47] ~ LibCURL_jll ⇒ v7.84.0+0
    Updating `C:\Users\boris\.julia\dev\GMT_jll\Manifest.toml`
  [692b3bcd] + JLLWrappers v1.4.1
  [21216c6a] + Preferences v1.3.0
  [8ce61222] + Arrow_jll v10.0.0+1
  [6e34b625] + Bzip2_jll v1.0.8+0
  [83423d85] + Cairo_jll v1.16.1+1
  [2e619515] + Expat_jll v2.4.8+0
  [b22a6f82] + FFMPEG_jll v4.4.2+2
  [f5851436] + FFTW_jll v3.3.10+0
  [a3f928ae] + Fontconfig_jll v2.13.93+0
  [d7e528f0] + FreeType2_jll v2.10.4+0
  [559328eb] + FriBidi_jll v1.0.10+0
  [a7073274] + GDAL_jll v301.600.200+0
  [d604d12d] + GEOS_jll v3.11.0+0
  [78b55507] + Gettext_jll v0.21.0+0
  [61579ee1] + Ghostscript_jll v9.55.0+4
  [7746bdde] + Glib_jll v2.74.0+2
  [3b182d85] + Graphite2_jll v1.3.14+0
  [0234f1f7] + HDF5_jll v1.12.2+2
  [2e76f6c2] + HarfBuzz_jll v2.8.1+1
  [aacddb02] + JpegTurbo_jll v2.1.91+0
  [b39eb1a6] + Kerberos_krb5_jll v1.19.3+0
  [c1c5ebd0] + LAME_jll v3.100.1+0
  [17f450c3] + LAPACK32_jll v3.10.0+0
  [88015f11] + LERC_jll v3.0.0+1
  [1d63c593] + LLVMOpenMP_jll v15.0.4+0
  [dd4b983a] + LZO_jll v2.10.1+0
  [08be9ffa] + LibPQ_jll v14.3.0+1
⌅ [e9f186c6] + Libffi_jll v3.2.2+1
  [d4300ac3] + Libgcrypt_jll v1.8.7+0
  [7add5ba3] + Libgpg_error_jll v1.42.0+0
  [94ce4f54] + Libiconv_jll v1.16.1+2
  [4b2f31a3] + Libmount_jll v2.35.0+0
  [89763e89] + Libtiff_jll v4.4.0+0
  [38a345b3] + Libuuid_jll v2.36.0+0
  [d3a379c0] + LittleCMS_jll v2.12.0+0
  [5ced341a] + Lz4_jll v1.9.3+0
  [7243133f] + NetCDF_jll v400.902.5+1
  [e7412a2a] + Ogg_jll v1.3.5+1
⌅ [656ef2d0] + OpenBLAS32_jll v0.3.17+0
  [643b3616] + OpenJpeg_jll v2.4.0+0
  [458c3c95] + OpenSSL_jll v1.1.20+0
  [91d4177d] + Opus_jll v1.3.2+0
  [2f80f16e] + PCRE_jll v8.44.0+0
  [58948b4f] + PROJ_jll v900.100.0+0
  [30392449] + Pixman_jll v0.40.1+0
  [76ed43ae] + SQLite_jll v3.41.0+0
  [e0b8ae26] + Thrift_jll v0.16.0+0
  [02c8fc9c] + XML2_jll v2.10.3+0
  [aed1982a] + XSLT_jll v1.1.34+0
  [4f6342f7] + Xorg_libX11_jll v1.6.9+4
  [0c0b7dd1] + Xorg_libXau_jll v1.0.9+4
  [a3789734] + Xorg_libXdmcp_jll v1.1.3+4
  [1082639a] + Xorg_libXext_jll v1.3.4+4
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.0
  [a4e569a6] + Tar v1.10.1
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v0.5.2+0
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.0+0
  [14a3606d] + MozillaCACerts_jll v2022.2.1
  [4536629a] + OpenBLAS_jll v0.3.20+0
  [efcefdf7] + PCRE2_jll v10.40.0+0
  [83775a58] + Zlib_jll v1.2.12+3
  [8e850b90] + libblastrampoline_jll v5.1.1+0
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0
        Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`
Precompiling project...
  1 dependency successfully precompiled in 2 seconds. 75 already precompiled.

check that the libraries can be loaded

julia> using LibCURL_jll, NetCDF_jll, GDAL_jll, FFTW_jll, PCRE_jll, LAPACK32_jll, OpenBLAS32_jll, FFMPEG_jll, Glib_jll,  Ghostscript_jll, CompilerSupportLibraries_jll, LLVMOpenMP_jll, GMT_jll

Run GMT:

julia> run(`$(GMT_jll.gmt())`)

        GMT - The Generic Mapping Tools, Version 6.5.0_19413d4_2023.03.03 [64-bit] [MP] [8 cores]
        (c) 1991-2023 The GMT Team (https://www.generic-mapping-tools.org/team.html).

        Supported in part by the US National Science Foundation (http://www.nsf.gov/)
        and volunteers from around the world.

        GMT is distributed under the GNU LGPL License (http://www.gnu.org/licenses/lgpl.html).
        Dependencies: netCDF, GDAL, PCRE, FFTW, LAPACK, ZLIB, Ghostscript, GraphicsMagick, FFmpeg.

usage: gmt [options]
       gmt <module name> [<module-options>]

options:
  --help              List descriptions of available GMT modules.
  --new-script[=L]    Write GMT modern mode script template to standard output.
                      Optionally specify bash|csh|batch [Default is current shell].
  --new-glue=name     Write C code for external supplements to glue them to GMT.
  --show-bindir       Show directory with GMT executables.
  --show-citation     Show the most recent citation for GMT.
  --show-classic      Show all classic module names.
  --show-classic-core Show all classic module names (core only).
  --show-cores        Show number of available cores.
  --show-datadir      Show directory/ies with user data.
  --show-dataserver   Show URL of the remote GMT data server.
  --show-dcw          Show the DCW data version used.
  --show-doi          Show the DOI for the current release.
  --show-gshhg        Show the GSHHG data version used.
  --show-library      Show path of the shared GMT library.-show-citation     Show the most recent citation for GMT.
  --show-classic      Show all classic module names.
  --show-classic-core Show all classic module names (core only).
  --show-cores        Show number of available cores.
  --show-datadir      Show directory/ies with user data.
  --show-dataserver   Show URL of the remote GMT data server.
  --show-dcw          Show the DCW data version used.
  --show-doi          Show the DOI for the current release.
  --show-gshhg        Show the GSHHG data version used.
  --show-library      Show path of the shared GMT library.
  --show-modules      Show all modern module names.
  --show-modules-core Show all modern module names (core only).
  --show-plugindir    Show directory for plug-ins.
  --show-sharedir     Show directory for shared GMT resources.
  --show-userdir      Show full path of user's ~/.gmt dir
  --version           Print GMT version number.
if <module-options> is '=' we call exit (0) if module exist and non-zero otherwise.

Run GMT with command line options:

julia> args="--version"
"--version"
julia> run(`$(GMT_jll.gmt()) $(args)`);
6.5.0_19413d4_2023.03.03

For completion once again my system;

julia> versioninfo()
Julia Version 1.8.3
Commit 0434deb161 (2022-11-14 20:14 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × 12th Gen Intel(R) Core(TM) i3-1215U
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, goldmont)
  Threads: 1 on 8 virtual cores

Perhaps you have a different system and it does not work on that one? As mentioned, it works on 1.9 as well (for me).

boriskaus · 2023-03-11T10:54:23Z

And do you know how to run the gmt executable with options? If it worked would be run(GMT_jll.gmt("--user-dir")). I need to capture the output of a couple of those options.

The option --user-dir does not do anything and is also not listed in the help of gmt. There is, however, the following option:

julia> args="--show-userdir"
"--show-userdir"

julia> run(`$(GMT_jll.gmt()) $(args)`);
C:/Users/boris/.gmt

you can catch the output of the above command into a string with;

julia> out = read(`$(GMT_jll.gmt()) $(args)`, String)
"C:/Users/boris/.gmt\n"
julia> out = readchomp(`$(GMT_jll.gmt()) $(args)`)
"C:/Users/boris/.gmt"

joa-quim · 2023-03-11T11:35:23Z

Good, thanks. So that must mean that the libblastrampoline-5-0-2.dll problem was solved upstream.

If you can load the GMT_jll branch, you should be able to actually run GMT.jl. It works but I already found one test that fails with the MinGW binaries but works with the MSVC ones.

julia> plot(rand(8,2), bg="-circ", Vd=2)
makecpt [ERROR]: Option -G: Unable to parse 0.25

vs

julia> plot(rand(8,2), bg="-circ", Vd=2)
2-element Vector{String}:
 "grdimage -D  -JX14c/9.5c -P -K > C:\\TEMP\\GMTjl_tmp.ps"
 "psxy  -JX14c/9.5c -Baf -BWSen -R0/0.88/0.06/0.86 -K -O >> C:\\TEMP\\GMTjl_tmp.ps"

boriskaus · 2023-03-11T11:45:28Z

I suppose I need to indicate in GMT.jl that I want to use the GMT_jll version instead of the current way of doing that.

If you can tell me exactly how to set that up and reproduce the plot command above I’m happy to try.

joa-quim · 2023-03-11T11:55:29Z

I think I saw that you can do ] add GMT.jl@GMT_jll or ] add GMT.jl#GMT_jll

Then it would only be to run the command I posted. But there are more and this doesn't look good. The errors come from the binaries themselves and to investigate them I need to debug bur for one side I would need to build them myself locally and for another I certainly do not want to use dbg. For me debugging is only with Visual Studio debugger.

julia> gmt("grdview -R-15/15/-15/15/-1/1 -JX6i/0 -Baf -Bza -n+a -p120/30 -Qs -JZ5 -Ctopo -I45 -P -K > lixo.ps", G)
grdview [ERROR]: Option -I: Must specify intensity file, value, or modifiers
ERROR: Something went wrong when calling the module. GMT error number = 72

boriskaus · 2023-03-11T15:12:35Z

yes, I can confirm that the tests above don´t work. It is encouraging, though, that quite a few of the tests do work.
Nevertheless I think it is easier to debug this once GMT_jll is registered at least once, as you can locally override the libraries with a local BB build.

joa-quim · 2023-03-11T16:29:30Z

The big shit is that, when running from CLI my MinGW build passes all tests of its equivalent MSVC build. But that proves little because the code paths followed by the CLI and wrapped versions (Julia, Matlab, Python) are not the same. It should be relatively easy if I could debug the gcc (MinGW) build, but again no way that I'll submit to the gdb path pain.

But I agree that this could be merged though not yet accepted as a Conda binaries replacement. Unless there is no problems with the Linux and MacOS binaries. Yet, before that the PROJ_jll dependency must be added.

joa-quim · 2023-03-11T17:24:10Z

One thing if you know about that. I see that the dependencies have

    Dependency("NetCDF_jll", compat="400.902.5")
    Dependency(PackageSpec(name="GDAL_jll", uuid="a7073274-a066-55f0-b90d-d619367d196c"); compat="301.600.200")

Not exactly sure how to convert those compat numbers to programs versions. I take it that it means

netcdf 4.9.x
gdal 3.6.x (what is that 301?)

but there is an elephant in the room which is the HDF5 versionupon which they linked againt. HDF5 is very, very picky with their versions. One cannot have a netCDF linked against 1.12.1 that is used to link in GDAL that itself was linked against 1.12.2. How are these things handled by BinaryBuilder?

joa-quim · 2023-03-11T18:35:10Z

There are good news though. My Linux (Ubuntu 20.04) build in Codespaces does not fail in those cases (though my WSL with Ubuntu 22.04 still errors with libcurl). But it errors

        GMTLOGO
psconvert [ERROR]: Cannot execute Ghostscript (gs).
GMT: Error During Test at /home/codespace/.julia/packages/GMT/p9huK/test/runtests.jl:5
  Got exception outside of a @test

clearly the gs executable is not being found, but Ghostscript_jll is a dependency.

boriskaus · 2023-03-11T22:18:04Z

One thing if you know about that. I see that the dependencies have
    Dependency("NetCDF_jll", compat="400.902.5")
    Dependency(PackageSpec(name="GDAL_jll", uuid="a7073274-a066-55f0-b90d-d619367d196c"); compat="301.600.200")
Not exactly sure how to convert those compat numbers to programs versions. I take it that it means

netcdf 4.9.x

gdal 3.6.x (what is that 301?)

but there is an elephant in the room which is the HDF5 versionupon which they linked againt. HDF5 is very, very picky with their versions. One cannot have a netCDF linked against 1.12.1 that is used to link in GDAL that itself was linked against 1.12.2. How are these things handled by BinaryBuilder?

You can see that in the build recipes for NetCDF:

upstream_version = v"4.9.0"

# Offset to add to the version number.  Remember to always bump this.
version_offset = v"0.2.5"

version = VersionNumber(upstream_version.major * 100 + version_offset.major,
                        upstream_version.minor * 100 + version_offset.minor,
                        upstream_version.patch * 100 + version_offset.patch)

I believe this is done because the BB version always need to increase if you add new options or dependencies to the BB build (even when the upstream version remains the same). Therefore it may quickly become out of whack with the real version number, so it is multiplied by 100 & incremented accordingly.

boriskaus · 2023-03-11T22:21:51Z

clearly the gs executable is not being found, but Ghostscript_jll is a dependency.

according to the build recipe of Ghostscript_jll, it should create a binary called gs:

  ExecutableProduct("gs", :gs),

Check whether that is available as Ghostscript_jll.gs()

Edit: Yes it is indeed available in Ghostscript_jll. If not done automatically, you have to make Ghostscript_jll a dependency of GMT.jl and can call it like you call the gmt binary.

joa-quim · 2023-03-14T20:49:03Z

OK, I have this working now on Linux but with a few strange errors on the Codespaces (where some test) fail but all pass in my WSL installation. Clearly we need more testing bur for that we also need this merged.

Note, on Windows the JLL is not used because of other strange errors that I'll need to find a way to debug them.

Boris, could you please rebuild the JLL using a more recent GMT commit? Several issues in GMT were fixed/improved since the last build.

boriskaus · 2023-03-14T22:44:53Z

Boris, could you please rebuild the JLL using a more recent GMT commit? Several issues in GMT were fixed/improved since the last build.

done (using the latest build of GMT that passed the CI tests).

boriskaus · 2023-03-15T09:13:35Z

@giordano: we believe this is now ok to be merged (if you're happy as well).

G/GMT/build_tarballs.jl

boriskaus added 4 commits July 21, 2022 18:55

first attempt to compile GMT. Appears to work on mac & linux, bnut no…

85abb34

…t on windows as the dependency NetCDF is not available on windows yet

don't think we need this

5d4eb33

more general NetCDF version (still doesn't like windows, though)

c61e973

undo

a51eb5d

giordano reviewed Jul 22, 2022

View reviewed changes

G/GMT/build_tarballs.jl Outdated Show resolved Hide resolved

G/GMT/build_tarballs.jl Outdated Show resolved Hide resolved

G/GMT/build_tarballs.jl Outdated Show resolved Hide resolved

boriskaus and others added 3 commits July 23, 2022 14:41

Update G/GMT/build_tarballs.jl

5211057

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

Update G/GMT/build_tarballs.jl

04e4d24

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

some progress with the windows install (still incomplete, though)

641d94c

giordano reviewed Jul 24, 2022

View reviewed changes

G/GMT/build_tarballs.jl Outdated Show resolved Hide resolved

giordano reviewed Jul 24, 2022

View reviewed changes

G/GMT/bundled/patches/win_mingw.patch Outdated Show resolved Hide resolved

G/GMT/bundled/patches/win_mingw.patch Outdated Show resolved Hide resolved

Update G/GMT/build_tarballs.jl

198c720

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

giordano reviewed Jul 24, 2022

View reviewed changes

G/GMT/build_tarballs.jl Outdated Show resolved Hide resolved

update version

bf360b1

boriskaus mentioned this pull request Jul 24, 2022

Use BinaryBuilder.jl to package dependencies GenericMappingTools/GMT.jl#447

Open

giordano mentioned this pull request Aug 24, 2022

Updated Gloo to version as of 2021-05-21 #5352

Merged

updated to use latest version (attempt to see if this compiles)

bdfde9d

Merge branch 'master' into GMT

9bd8147

add PROJ_jll dependency

7642321

Update build_tarballs.jl

7355ba2

update git hash

2b38c54

giordano reviewed Mar 15, 2023

View reviewed changes

G/GMT/build_tarballs.jl Outdated Show resolved Hide resolved

G/GMT/build_tarballs.jl Outdated Show resolved Hide resolved

G/GMT/build_tarballs.jl Outdated Show resolved Hide resolved

Apply suggestions from code review

4c4171e

giordano merged commit 0bb6989 into JuliaPackaging:master Mar 15, 2023

GMT #5199

GMT #5199

Conversation

boriskaus commented Jul 21, 2022 • edited Loading

boriskaus commented Jul 23, 2022

giordano commented Jul 23, 2022

giordano commented Jul 23, 2022

boriskaus commented Jul 24, 2022

giordano commented Jul 24, 2022

giordano commented Jul 24, 2022

boriskaus commented Jul 24, 2022

joa-quim commented Mar 2, 2023

giordano commented Mar 2, 2023

joa-quim commented Mar 2, 2023 • edited Loading

giordano commented Mar 2, 2023

joa-quim commented Mar 2, 2023

giordano commented Mar 2, 2023

joa-quim commented Mar 2, 2023

giordano commented Mar 7, 2023

joa-quim commented Mar 8, 2023

giordano commented Mar 8, 2023

joa-quim commented Mar 8, 2023

joa-quim commented Mar 8, 2023

boriskaus commented Mar 10, 2023

joa-quim commented Mar 10, 2023

boriskaus commented Mar 10, 2023 • edited Loading

joa-quim commented Mar 10, 2023

boriskaus commented Mar 11, 2023 • edited Loading

boriskaus commented Mar 11, 2023 • edited Loading

joa-quim commented Mar 11, 2023

boriskaus commented Mar 11, 2023

joa-quim commented Mar 11, 2023

boriskaus commented Mar 11, 2023

joa-quim commented Mar 11, 2023

joa-quim commented Mar 11, 2023

joa-quim commented Mar 11, 2023

boriskaus commented Mar 11, 2023 • edited Loading

boriskaus commented Mar 11, 2023 • edited Loading

joa-quim commented Mar 14, 2023

boriskaus commented Mar 14, 2023

boriskaus commented Mar 15, 2023

boriskaus commented Jul 21, 2022 •

edited

Loading

joa-quim commented Mar 2, 2023 •

edited

Loading

boriskaus commented Mar 10, 2023 •

edited

Loading

boriskaus commented Mar 11, 2023 •

edited

Loading

boriskaus commented Mar 11, 2023 •

edited

Loading

boriskaus commented Mar 11, 2023 •

edited

Loading

boriskaus commented Mar 11, 2023 •

edited

Loading