Detecting shared libraries on Windows with MinGW #602
Replies: 10 comments 24 replies
-
|
The best way to figure out what's going on is if we can somehow get access to meson-log.txt, although I think pip deleted everything on failure so that can be a bit confusing. Try passing Off the top of my head I think meson will automatically handle any libpath specified in link_args via a machine file or via $LDFLAGS. It's also possible to install pkg-config on Windows too, and let meson use that to detect gmp as a dependency instead of via find_library -- gmp.pc should be created because the autotools build on MinGW treats it as a kind of Unix and doesn't refuse to produce outputs it doesn't think are "interesting" on Windows (some projects have exactly this sort of manual logic via ifdefs for the msvc compiler). |
Beta Was this translation helpful? Give feedback.
-
|
Using Another way to keep using |
Beta Was this translation helpful? Give feedback.
-
|
Hello @oscarbenjamin. AFAICT, the issue you report does not have much to do with meson-python, but it is purely a Meson issue. meson-python is only the bridge between the Python build frontend and Meson, it is not involved in dependencies resolution. I'm saying this because none of the meson-python developers are very familiar with Windows, thus you may get better help on the Meson help channels. Indeed, when debugging this kind of issue, the first thing I recommend is to move cibuildwheel and meson-python out of the loop. Two less levels of indirection helps in getting to the cause of the problem faster. In the specific case, don't gmp, mpfr, and flint provide pkg-config files? To debug how Meson is trying to look up the dependencies, you need to look into the detailed Meson log file (pointed to in the last line of the console output you included). However, to get at it, you need to avoid pip deleting the temporary build directory. If you are stuck at the |
Beta Was this translation helpful? Give feedback.
-
|
Thanks all. I don't use Windows myself either and in fact I won't have access to a Windows machine for a few days at least so this is currently a debug via CI situation... Switching to pkg-config seems like the best idea and in keeping with how meson is supposed to work. I am testing with pkg-config now in flintlib/python-flint#129. Building wheels is working for OSX and Linux but failing on Windows with: I guess there is something wrong with the MinGW environment that I just need to figure out or perhaps I am installing the wrong package... A new problem now though is that if I do $ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip install .
Processing /Users/enojb/work/dev/python-flint
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: python-flint
Building wheel for python-flint (pyproject.toml) ... done
Created wheel for python-flint: filename=python_flint-0.6.0-cp312-cp312-macosx_14_0_arm64.whl size=1728563 sha256=65afc605ec9b90908b5bd2426887eabb052c83dc0a3f6c0abb5e3b44ec97140a
Stored in directory: /Users/enojb/Library/Caches/pip/wheels/dc/58/41/99d653384774c5a6090230bd9df1409f0209f328299de41900
Successfully built python-flint
Installing collected packages: python-flint
Successfully installed python-flint-0.6.0
$ DYLD_LIBRARY_PATH=$(pwd)/.local/lib python -m flint
Traceback (most recent call last):
File "<frozen runpy>", line 189, in _run_module_as_main
File "<frozen runpy>", line 148, in _get_module_details
File "<frozen runpy>", line 112, in _get_module_details
File "/Users/enojb/.pyenv/versions/python-flint-3.12.git/lib/python3.12/site-packages/flint/__init__.py", line 1, in <module>
from .pyflint import *
File "pyflint.pyx", line 1, in init flint.pyflint
File "flint_base.pyx", line 1, in init flint.flint_base.flint_base
ImportError: dlopen(/Users/enojb/.pyenv/versions/python-flint-3.12.git/lib/python3.12/site-packages/flint/flint_base/flint_context.cpython-312-darwin.so, 0x0002): symbol not found in flat namespace '_flint_cleanup'More strangely in CI the same operation seems to work or not depending on the version of Flint... Somehow I guess that using pkg-config and |
Beta Was this translation helpful? Give feedback.
-
|
GitHub Actions on Windows installs a Perl version that ships its own totally broken, outdated, binary-incompatible copies of GCC, pkg-config, and various third-party libraries which perl internally uses, such as zlib. They do this because they want users to have a perl.exe so they can run perl programs in CI. Perl just sticks every bit of software it can onto PATH because they'd rather break your builds as long as cpan users can install cpan packages without setting up the perl environment -- and apparently that somehow also means cpan can't be a wrapper program that sets up the perl environment before running the actual cpan program. Depending on the precise order of PATH, the msys2 that you wanted to use, or the mingw from perl, may or may not get priority. Based on the log, it's somehow finding Perl when it shouldn't. |
Beta Was this translation helpful? Give feedback.
-
|
Here's how to install I have a half-implemented thing to build |
Beta Was this translation helpful? Give feedback.
-
Thanks. This has gotten me one step closer. Now the build fails with: So it looks like the chocolatey pkg-config was found and meson is happy to use it. However it does not find gmp. This is all in CI for now so I added a few $ ls .local/lib
libgmp.dll.a
libgmp.la
pkgconfig
$ ls .local/bin
libgmp-10.dll
$ ls .local/lib/pkgconfig
gmp.pc
$ cat .local/lib/pkgconfig/gmp.pc
prefix=/d/a/python-flint/python-flint/.local
exec_prefix=${prefix}
includedir=${prefix}/include
libdir=${exec_prefix}/lib
Name: GNU MP
Description: GNU Multiple Precision Arithmetic Library
URL: https://gmplib.org
Version: 6.3.0
Cflags: -I${includedir}
Libs: -L${libdir} -lgmpI have the environment variable The main difference I see for Windows is that the dll is in the I am not really sure how I can get the meson log out of CI. |
Beta Was this translation helpful? Give feedback.
-
It doesn't, since that is not how it's supposed to be laid out, and Ideally, GMP fixes that so it doesn't get built in an inconsistent state to begin with of course. |
Beta Was this translation helpful? Give feedback.
-
|
If you pass the previously mentioned -C builddir=./builddir argument to build the wheel, then meson-python should end up using a build directory that is NOT the one pip auto-deletes. At that point, you can have the CI machine |
Beta Was this translation helpful? Give feedback.
-
I'm trying to think how to do this. It is not trivial because I'm running via cibuildwheel so I would need to get cibuildwheel to pass an option to the mesonpy build backend. It is probably easier to stop using cibuildwheel but that also would take a bunch of careful rewriting. There are a lot interdependent steps that are currently all controlled by cibuildwheel... |
Beta Was this translation helpful? Give feedback.
-
|
https://cibuildwheel.pypa.io/en/stable/options/#build-frontend It looks like you can tell cibuildwheel to use "pip, with such and such additional pip arguments". Does that help? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, that does help. I'm trying now... Actually one thing that meson-python could have that would be very helpful in this situation is an easy way to get it to show the full logs like a |
Beta Was this translation helpful? Give feedback.
-
I'm not sure about this. All three libraries put the DLL files in $ ls .local/bin
libflint.19.0.0.dll
libflint.19.dll
libflint.dll
libgmp-10.dll
libmpfr-6.dll
$ ls .local/lib
libflint.dll.a
libgmp.dll.a
libgmp.la
libmpfr.dll.a
libmpfr.la
pkgconfig
$ ls .local/lib/pkginfo
flint.pc
gmp.pc
mpfr.pcIt seems that the convention is that Note that I build these dependencies the same way when building with setuptools and building with setuptools works on Windows with MinGW as well as OSX and Linux. I am in any case testing what happens if I copy the DLL into lib. Just copying |
Beta Was this translation helpful? Give feedback.
-
|
🤔 okay maybe my relatively limited Windows knowledge made me give the wrong suggestion here. |
Beta Was this translation helpful? Give feedback.
-
|
This comment got hidden above so I am going to repeat it here because I think it is worth seeing:
|
Beta Was this translation helpful? Give feedback.
-
|
Adding it to #246 |
Beta Was this translation helpful? Give feedback.
-
|
mesonbuild/meson@master...eli-schwartz:ci-fold Seems I have a branch for it... I must have been thinking about this in the past. |
Beta Was this translation helpful? Give feedback.
-
|
Okay, here is the log!: Click to expand |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure what the log might show but it looks like the relevant part is From there the possibly significant thing I notice is I have set this environment variable in cibuildwheel configuration in pyproject.toml: It looks as if my setting for the environment variable might have been overridden somehow... |
Beta Was this translation helpful? Give feedback.
-
So maybe the method you used to set EDIT: and that script is hooked in like so: |
Beta Was this translation helpful? Give feedback.
-
It was working for OSX and Linux (before I butchered everything to try to get more info out of the Windows CI). Maybe it somehow doesn't work for Windows though because something else overrides it but only in the Windows case. I'll try setting the environment variable some other way.
I'm very much not a shell scripting expert but isn't that setting a local variable? Do you not need |
Beta Was this translation helpful? Give feedback.
-
|
Oh sorry, wrong line linked. Here's the exports in the
It's all not pretty, but it has worked reliably for a while. |
Beta Was this translation helpful? Give feedback.
-
I've followed the same and it looks like it's working! I have wheels built for Windows, Linux and OSX that seem to work. There is a problem with older Flint versions that I will investigate but otherwise all good! One mystery I have not yet resolved is what it is that is overriding the Thanks all for your help! |
Beta Was this translation helpful? Give feedback.
-
I don't think it is cibuildwheel or meson-python doing this but it could be meson: |
Beta Was this translation helpful? Give feedback.
-
Yes, I think that should work automatically.
If you want to support that out of the box you can do something like: gmp_dep = dependency('gmp', required: false)
if not gmp_dep.found()
# We support using MPIR instead of GMP, so try that as a fallback
gmp_dep = dependency('mpir', required: false)
endif
if not gmp_dep.found()
error('Neither GMP nor MPIR found ...')
endifOr if you don't like it / don't want to support it directly, you require conda-forge to carry a patch for it. |
Beta Was this translation helpful? Give feedback.
-
|
We should probably ask for some feedback on flintlib/python-flint#129 from conda folks (CC @isuruf) before putting out a new release. I think that linking with MPIR would be best handled as an explicit option rather than a dependency fallback. Something like: # meson_options.txt
option('use-mpir', type: 'boolean', value: false,
description: 'Link with MPIR instead of GMP')
# meson.build
if not get_option('use-mpir')
gmp_dep = dependency('gmp')
else
gmp_dep = dependency('mpir')
endifThen conda-forge does not need to patch anything and can just do python -m build -Csetup-args="-Duse-mpir=true" |
Beta Was this translation helpful? Give feedback.
-
|
Well, since you mention mpir.... A couple weeks ago I gave the last rites to mpir in Gentoo: https://packages.gentoo.org/packages/sci-libs/mpir It is outdated and unmaintained, which isn't new information, but the situation has now gotten so bad that it doesn't successfully configure with modern compilers at all. |
Beta Was this translation helpful? Give feedback.
-
|
I don't know how easy it is for conda-forge to switch from MPIR to GMP. GMP cannot be built with runtime CPU detection ( |
Beta Was this translation helpful? Give feedback.
-
|
It looks like Flint 3.1.0 has removed the |
Beta Was this translation helpful? Give feedback.
-
Following on from #410 I am now attempting to convert python-flint's build system from setuptools to meson-python in flintlib/python-flint#129 (I am not attempting to do what was described in that discussion though). So far I have everything working to build wheels for OSX and Linux but the Windows build fails because meson-python cannot find the external shared library dependencies that we build beforehand.
The final part of the CI log is:
The way that this build works is that we first build GMP, MPFR and Flint which are all autotools-based. We install these into a build-local directory like:
./configure --prefix=$(pwd)/.local make make installOn Windows this is done with msys and MinGW.
Then we run cibuildwheel but set the environment variables
LIBRARY_PATHetc to include the.local/libetc directories:https://github.com/flintlib/python-flint/blob/1ce152dffc356af69b1d4c2ea0eb08854f3d733b/pyproject.toml#L31-L35
With the current setuptools build this works on all operating systems and with Windows and MinGW in particular. With the meson-python configuration that I have so far this works on OSX and Linux but fails on Windows.
It looks like meson-python has correctly found the MinGW compiler but fails to find the
libgmp.dllthat we built. This is how the top-levelmeson.buildlooks:One thing I know that differs between Windows and Unix is that .dlls are searched on
PATHso I tried adding some directories to that as well but it did not help.Is there perhaps something that should be used instead of environment variables to make this work?
Should it be expected that
meson-pythonwould find these shared libraries in this situation (given that I think MinGW would be able to find them)?Another possibility in future is that we might use MSVC instead of MinGW for building the extension modules but we still need to use MinGW for the dependencies (for GMP in particular) so I assume that we will have the same problem somehow if using MSVC.
Beta Was this translation helpful? Give feedback.
All reactions