Building internal modules with the cross build is (slightly) broken #162
Comments
lazka
changed the title
|
I've had a look at this now, and I think I understand most aspects of it. For some additional context - I started digging into this, because when I tested builds with llvm-mingw, where I used LTO for some base toolchain files, my builds of cpython-mingw crashed at runtime. I think I have an explanation to this. When linking these shared modules, These shared libraries are built and linked by running However for mingw targets, this is different, since each of the shared modules I can avoid it when building python within llvm-mingw with a modification to https://github.com/mstorsjo/llvm-mingw/blob/master/build-python.sh like this: diff --git a/build-python.sh b/build-python.sh
index f6ddf25..51b832d 100755
--- a/build-python.sh
+++ b/build-python.sh
@@ -155,6 +155,7 @@ export CXX=$HOST-g++
--without-ensurepip \
--without-c-locale-coercion
+$MAKE -j$CORES libpython3.11.dll.a
$MAKE -j$CORES
$MAKE install
rm -rf $PREFIX/lib/python*/testThis isn't very pretty though. It can also be worked around directly in cpython-mingw's makefile like this: diff --git a/Makefile.pre.in b/Makefile.pre.in
index 93728d6224..32433e67e3 100644
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -778,7 +778,7 @@ $(srcdir)/Modules/_blake2/blake2s_impl.c: $(srcdir)/Modules/_blake2/blake2b_impl
# -s, --silent or --quiet is always the first char.
# Under BSD make, MAKEFLAGS might be " -s -v x=y".
# Ignore macros passed by GNU make, passed after --
-sharedmods: $(PYTHON_FOR_BUILD_DEPS) pybuilddir.txt @LIBMPDEC_INTERNAL@ @LIBEXPAT_INTERNAL@
+sharedmods: $(PYTHON_FOR_BUILD_DEPS) pybuilddir.txt $(LDLIBRARY) @LIBMPDEC_INTERNAL@ @LIBEXPAT_INTERNAL@
@case "`echo X $$MAKEFLAGS | sed 's/^X //;s/ -- .*//'`" in \
*\ -s*|s*) quiet="-q";; \
*) quiet="";; \I.e. add a dependency on With regards to correctness, this is an unnecessary restriction if building for e.g. Linux, because So, in most cases, this has been a benign issue, that mostly has caused confusion when reading build logs. However, it can actually cause broken builds of python, that crash at runtime. When linking a
Most modules fail to link with an error like this, but 4 modules actually link successfully; the modules Thus, this build race condition, can in some cases cause a broken build, if we hit this condition:
In most cases, So by fixing this missing dependency, we would avoid this race condition and this build nondeterminism. |
|
Thanks for the investigation! Should I create a PR based on it? |
I guess I could create one myself as well; the main question is, is it ok to add a dependency on |
|
I think it's fine as long as the commit message mentions it. Our fork currently doesn't work for Linux (see #131) so many of the Makefile changes aren't portable right now and would need similar adjustments.
it doesn't, officially at least |
Ok, thanks, that's good context info. Yeah ideally things would be in an upstreamable form, in case upstream would care to accept such patches, but I understand the realities of things :-)
Oh, ok. The upstream makefile at least has some mentions of it, but I guess it's not maintained: https://github.com/python/cpython/blob/v3.12.0/Makefile.pre.in#L868-L875 |
mstorsjo
added a commit
to mstorsjo/cpython-mingw
that referenced
this issue
Dec 4, 2023
When building shared modules (*.pyd) for MinGW, they link against the import library for the main python (e.g. -lpython3.11). When linking shared modules on Linux, this doesn't happen, but those shared libraries are linked with undefined references which then are fulfilled by the host process when they are loaded. Add a dependency to make sure that $(LDLIBRARY), e.g. libpython$(LDVERSION).dll.a, exists before starting to build the shared modules. Previously, if libpython$(LDVERSION).dll.a didn't exist when the shared modules were linked, some of them failed to link, but this wasn't reported as an error in the build, and a later invocation of setup.py would retry building them, which then would succeed. E.g. there was a seemingly benign race condition in the build. However, in some rare cases, the race condition no longer was benign. The build also produces a corresponding static library, $(LIBRARY), e.g. libpython$(VERSION)$(ABIFLAGS).a - which in the end would be named similarly, libpython3.11.a, when $(LDLIBRARY) would be libpython3.11.dll.a. If the static library exists but the import library doesn't, then most modules would still fail to link, but a few ones wouldn't (select, _multiprocessing, _overlapped and _socket). The ones that did succeed linking with the static library would crash at runtime though. By making sure that the right, intended library to fulfill the linker parameter "-lpython3.11" exists before building the shared modules, we avoid the whole race condition that in some rare cases could produce a Python build that crashes at runtime. For Linux targets, this extra dependency is unnecessary, but should not cause other problems ($(LDLIBRARY) is set to the same as $(LIBRARY) if not building any shared library). This fixes MSYS2 cpython-mingw issue msys2-contrib#162.
lazka
pushed a commit
that referenced
this issue
Dec 4, 2023
When building shared modules (*.pyd) for MinGW, they link against the import library for the main python (e.g. -lpython3.11). When linking shared modules on Linux, this doesn't happen, but those shared libraries are linked with undefined references which then are fulfilled by the host process when they are loaded. Add a dependency to make sure that $(LDLIBRARY), e.g. libpython$(LDVERSION).dll.a, exists before starting to build the shared modules. Previously, if libpython$(LDVERSION).dll.a didn't exist when the shared modules were linked, some of them failed to link, but this wasn't reported as an error in the build, and a later invocation of setup.py would retry building them, which then would succeed. E.g. there was a seemingly benign race condition in the build. However, in some rare cases, the race condition no longer was benign. The build also produces a corresponding static library, $(LIBRARY), e.g. libpython$(VERSION)$(ABIFLAGS).a - which in the end would be named similarly, libpython3.11.a, when $(LDLIBRARY) would be libpython3.11.dll.a. If the static library exists but the import library doesn't, then most modules would still fail to link, but a few ones wouldn't (select, _multiprocessing, _overlapped and _socket). The ones that did succeed linking with the static library would crash at runtime though. By making sure that the right, intended library to fulfill the linker parameter "-lpython3.11" exists before building the shared modules, we avoid the whole race condition that in some rare cases could produce a Python build that crashes at runtime. For Linux targets, this extra dependency is unnecessary, but should not cause other problems ($(LDLIBRARY) is set to the same as $(LIBRARY) if not building any shared library). This fixes MSYS2 cpython-mingw issue #162.
|
Thanks for the merge! I believe this issue can be closed now? |
lazka
pushed a commit
that referenced
this issue
Dec 6, 2023
When building shared modules (*.pyd) for MinGW, they link against the import library for the main python (e.g. -lpython3.11). When linking shared modules on Linux, this doesn't happen, but those shared libraries are linked with undefined references which then are fulfilled by the host process when they are loaded. Add a dependency to make sure that $(LDLIBRARY), e.g. libpython$(LDVERSION).dll.a, exists before starting to build the shared modules. Previously, if libpython$(LDVERSION).dll.a didn't exist when the shared modules were linked, some of them failed to link, but this wasn't reported as an error in the build, and a later invocation of setup.py would retry building them, which then would succeed. E.g. there was a seemingly benign race condition in the build. However, in some rare cases, the race condition no longer was benign. The build also produces a corresponding static library, $(LIBRARY), e.g. libpython$(VERSION)$(ABIFLAGS).a - which in the end would be named similarly, libpython3.11.a, when $(LDLIBRARY) would be libpython3.11.dll.a. If the static library exists but the import library doesn't, then most modules would still fail to link, but a few ones wouldn't (select, _multiprocessing, _overlapped and _socket). The ones that did succeed linking with the static library would crash at runtime though. By making sure that the right, intended library to fulfill the linker parameter "-lpython3.11" exists before building the shared modules, we avoid the whole race condition that in some rare cases could produce a Python build that crashes at runtime. For Linux targets, this extra dependency is unnecessary, but should not cause other problems ($(LDLIBRARY) is set to the same as $(LIBRARY) if not building any shared library). This fixes MSYS2 cpython-mingw issue #162.
lazka
pushed a commit
that referenced
this issue
Dec 7, 2023
When building shared modules (*.pyd) for MinGW, they link against the import library for the main python (e.g. -lpython3.11). When linking shared modules on Linux, this doesn't happen, but those shared libraries are linked with undefined references which then are fulfilled by the host process when they are loaded. Add a dependency to make sure that $(LDLIBRARY), e.g. libpython$(LDVERSION).dll.a, exists before starting to build the shared modules. Previously, if libpython$(LDVERSION).dll.a didn't exist when the shared modules were linked, some of them failed to link, but this wasn't reported as an error in the build, and a later invocation of setup.py would retry building them, which then would succeed. E.g. there was a seemingly benign race condition in the build. However, in some rare cases, the race condition no longer was benign. The build also produces a corresponding static library, $(LIBRARY), e.g. libpython$(VERSION)$(ABIFLAGS).a - which in the end would be named similarly, libpython3.11.a, when $(LDLIBRARY) would be libpython3.11.dll.a. If the static library exists but the import library doesn't, then most modules would still fail to link, but a few ones wouldn't (select, _multiprocessing, _overlapped and _socket). The ones that did succeed linking with the static library would crash at runtime though. By making sure that the right, intended library to fulfill the linker parameter "-lpython3.11" exists before building the shared modules, we avoid the whole race condition that in some rare cases could produce a Python build that crashes at runtime. For Linux targets, this extra dependency is unnecessary, but should not cause other problems ($(LDLIBRARY) is set to the same as $(LIBRARY) if not building any shared library). This fixes MSYS2 cpython-mingw issue #162.
lazka
pushed a commit
that referenced
this issue
Feb 10, 2024
When building shared modules (*.pyd) for MinGW, they link against the import library for the main python (e.g. -lpython3.11). When linking shared modules on Linux, this doesn't happen, but those shared libraries are linked with undefined references which then are fulfilled by the host process when they are loaded. Add a dependency to make sure that $(LDLIBRARY), e.g. libpython$(LDVERSION).dll.a, exists before starting to build the shared modules. Previously, if libpython$(LDVERSION).dll.a didn't exist when the shared modules were linked, some of them failed to link, but this wasn't reported as an error in the build, and a later invocation of setup.py would retry building them, which then would succeed. E.g. there was a seemingly benign race condition in the build. However, in some rare cases, the race condition no longer was benign. The build also produces a corresponding static library, $(LIBRARY), e.g. libpython$(VERSION)$(ABIFLAGS).a - which in the end would be named similarly, libpython3.11.a, when $(LDLIBRARY) would be libpython3.11.dll.a. If the static library exists but the import library doesn't, then most modules would still fail to link, but a few ones wouldn't (select, _multiprocessing, _overlapped and _socket). The ones that did succeed linking with the static library would crash at runtime though. By making sure that the right, intended library to fulfill the linker parameter "-lpython3.11" exists before building the shared modules, we avoid the whole race condition that in some rare cases could produce a Python build that crashes at runtime. For Linux targets, this extra dependency is unnecessary, but should not cause other problems ($(LDLIBRARY) is set to the same as $(LIBRARY) if not building any shared library). This fixes MSYS2 cpython-mingw issue #162.
lazka
pushed a commit
that referenced
this issue
Feb 12, 2024
When building shared modules (*.pyd) for MinGW, they link against the import library for the main python (e.g. -lpython3.11). When linking shared modules on Linux, this doesn't happen, but those shared libraries are linked with undefined references which then are fulfilled by the host process when they are loaded. Add a dependency to make sure that $(LDLIBRARY), e.g. libpython$(LDVERSION).dll.a, exists before starting to build the shared modules. Previously, if libpython$(LDVERSION).dll.a didn't exist when the shared modules were linked, some of them failed to link, but this wasn't reported as an error in the build, and a later invocation of setup.py would retry building them, which then would succeed. E.g. there was a seemingly benign race condition in the build. However, in some rare cases, the race condition no longer was benign. The build also produces a corresponding static library, $(LIBRARY), e.g. libpython$(VERSION)$(ABIFLAGS).a - which in the end would be named similarly, libpython3.11.a, when $(LDLIBRARY) would be libpython3.11.dll.a. If the static library exists but the import library doesn't, then most modules would still fail to link, but a few ones wouldn't (select, _multiprocessing, _overlapped and _socket). The ones that did succeed linking with the static library would crash at runtime though. By making sure that the right, intended library to fulfill the linker parameter "-lpython3.11" exists before building the shared modules, we avoid the whole race condition that in some rare cases could produce a Python build that crashes at runtime. For Linux targets, this extra dependency is unnecessary, but should not cause other problems ($(LDLIBRARY) is set to the same as $(LIBRARY) if not building any shared library). This fixes MSYS2 cpython-mingw issue #162.
lazka
pushed a commit
that referenced
this issue
Apr 11, 2024
When building shared modules (*.pyd) for MinGW, they link against the import library for the main python (e.g. -lpython3.11). When linking shared modules on Linux, this doesn't happen, but those shared libraries are linked with undefined references which then are fulfilled by the host process when they are loaded. Add a dependency to make sure that $(LDLIBRARY), e.g. libpython$(LDVERSION).dll.a, exists before starting to build the shared modules. Previously, if libpython$(LDVERSION).dll.a didn't exist when the shared modules were linked, some of them failed to link, but this wasn't reported as an error in the build, and a later invocation of setup.py would retry building them, which then would succeed. E.g. there was a seemingly benign race condition in the build. However, in some rare cases, the race condition no longer was benign. The build also produces a corresponding static library, $(LIBRARY), e.g. libpython$(VERSION)$(ABIFLAGS).a - which in the end would be named similarly, libpython3.11.a, when $(LDLIBRARY) would be libpython3.11.dll.a. If the static library exists but the import library doesn't, then most modules would still fail to link, but a few ones wouldn't (select, _multiprocessing, _overlapped and _socket). The ones that did succeed linking with the static library would crash at runtime though. By making sure that the right, intended library to fulfill the linker parameter "-lpython3.11" exists before building the shared modules, we avoid the whole race condition that in some rare cases could produce a Python build that crashes at runtime. For Linux targets, this extra dependency is unnecessary, but should not cause other problems ($(LDLIBRARY) is set to the same as $(LIBRARY) if not building any shared library). This fixes MSYS2 cpython-mingw issue #162.
lazka
pushed a commit
that referenced
this issue
Apr 12, 2024
When building shared modules (*.pyd) for MinGW, they link against the import library for the main python (e.g. -lpython3.11). When linking shared modules on Linux, this doesn't happen, but those shared libraries are linked with undefined references which then are fulfilled by the host process when they are loaded. Add a dependency to make sure that $(LDLIBRARY), e.g. libpython$(LDVERSION).dll.a, exists before starting to build the shared modules. Previously, if libpython$(LDVERSION).dll.a didn't exist when the shared modules were linked, some of them failed to link, but this wasn't reported as an error in the build, and a later invocation of setup.py would retry building them, which then would succeed. E.g. there was a seemingly benign race condition in the build. However, in some rare cases, the race condition no longer was benign. The build also produces a corresponding static library, $(LIBRARY), e.g. libpython$(VERSION)$(ABIFLAGS).a - which in the end would be named similarly, libpython3.11.a, when $(LDLIBRARY) would be libpython3.11.dll.a. If the static library exists but the import library doesn't, then most modules would still fail to link, but a few ones wouldn't (select, _multiprocessing, _overlapped and _socket). The ones that did succeed linking with the static library would crash at runtime though. By making sure that the right, intended library to fulfill the linker parameter "-lpython3.11" exists before building the shared modules, we avoid the whole race condition that in some rare cases could produce a Python build that crashes at runtime. For Linux targets, this extra dependency is unnecessary, but should not cause other problems ($(LDLIBRARY) is set to the same as $(LIBRARY) if not building any shared library). This fixes MSYS2 cpython-mingw issue #162.
lazka
pushed a commit
that referenced
this issue
Sep 10, 2024
When building shared modules (*.pyd) for MinGW, they link against the import library for the main python (e.g. -lpython3.11). When linking shared modules on Linux, this doesn't happen, but those shared libraries are linked with undefined references which then are fulfilled by the host process when they are loaded. Add a dependency to make sure that $(LDLIBRARY), e.g. libpython$(LDVERSION).dll.a, exists before starting to build the shared modules. Previously, if libpython$(LDVERSION).dll.a didn't exist when the shared modules were linked, some of them failed to link, but this wasn't reported as an error in the build, and a later invocation of setup.py would retry building them, which then would succeed. E.g. there was a seemingly benign race condition in the build. However, in some rare cases, the race condition no longer was benign. The build also produces a corresponding static library, $(LIBRARY), e.g. libpython$(VERSION)$(ABIFLAGS).a - which in the end would be named similarly, libpython3.11.a, when $(LDLIBRARY) would be libpython3.11.dll.a. If the static library exists but the import library doesn't, then most modules would still fail to link, but a few ones wouldn't (select, _multiprocessing, _overlapped and _socket). The ones that did succeed linking with the static library would crash at runtime though. By making sure that the right, intended library to fulfill the linker parameter "-lpython3.11" exists before building the shared modules, we avoid the whole race condition that in some rare cases could produce a Python build that crashes at runtime. For Linux targets, this extra dependency is unnecessary, but should not cause other problems ($(LDLIBRARY) is set to the same as $(LIBRARY) if not building any shared library). This fixes MSYS2 cpython-mingw issue #162.
lazka
pushed a commit
that referenced
this issue
Sep 10, 2024
When building shared modules (*.pyd) for MinGW, they link against the import library for the main python (e.g. -lpython3.11). When linking shared modules on Linux, this doesn't happen, but those shared libraries are linked with undefined references which then are fulfilled by the host process when they are loaded. Add a dependency to make sure that $(LDLIBRARY), e.g. libpython$(LDVERSION).dll.a, exists before starting to build the shared modules. Previously, if libpython$(LDVERSION).dll.a didn't exist when the shared modules were linked, some of them failed to link, but this wasn't reported as an error in the build, and a later invocation of setup.py would retry building them, which then would succeed. E.g. there was a seemingly benign race condition in the build. However, in some rare cases, the race condition no longer was benign. The build also produces a corresponding static library, $(LIBRARY), e.g. libpython$(VERSION)$(ABIFLAGS).a - which in the end would be named similarly, libpython3.11.a, when $(LDLIBRARY) would be libpython3.11.dll.a. If the static library exists but the import library doesn't, then most modules would still fail to link, but a few ones wouldn't (select, _multiprocessing, _overlapped and _socket). The ones that did succeed linking with the static library would crash at runtime though. By making sure that the right, intended library to fulfill the linker parameter "-lpython3.11" exists before building the shared modules, we avoid the whole race condition that in some rare cases could produce a Python build that crashes at runtime. For Linux targets, this extra dependency is unnecessary, but should not cause other problems ($(LDLIBRARY) is set to the same as $(LIBRARY) if not building any shared library). This fixes MSYS2 cpython-mingw issue #162.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Looking at https://github.com/msys2-contrib/cpython-mingw/actions/runs/6802267164/job/18495001638
It tries to build
_contextvars.cp311-mingw_x86_64_clang.pydbut fails with:lld: error: unable to find library -lpython3.11
then tries again at a later point and succeeds (likely because the dependency then exists)
The text was updated successfully, but these errors were encountered: