Copy libs/pythonXY.lib to support building of C extensions without distutils #1023

jcfr · 2017-02-23T22:47:48Z

This issue was discovered while using virtualenv to create an
isolated environment and test that a source distribution would compile
without using the knowledge hardcoded in distutils.sysconfig and
virtualenv/virtualenv_embedded/distutils-init.py

In other word, without relying on the fact virtualenv monkey patch distutils
to ensure the value associated with LIBDIR variable is correct.

This is relevant because the library location is implicitly specified
in PC/pyconfig.h

[...]

/* For an MSVC DLL, we can nominate the .lib files used by extensions */
                        /* So MSVC users need not specify the .lib file in
                        their Makefile (other compilers are generally
                        taken care of by distutils.) */

[...]

References:

macisamuele · 2018-08-10T17:46:46Z

Is there any interest on merging this PR?
I'm currently working on a small project with rust bindings on Windows and I'm not able to build it within a virtual environment. Applying those changes on top of the latest version of virtualenv I can finally build them.

Pinging @dstufft as one of the project moderator recently active.
If you think that this PR could be "mergeable" but it needs testing etc. let me know and I'll try to give some support

jcfr · 2018-08-10T17:58:52Z

@macisamuele ~~Here is a rebased topic, if it works well, I will push force this topic.~~ EDIT: This topic has been updated.

Thanks for your help

It would indeed be nice to get this integrated.

macisamuele · 2018-08-10T18:05:01Z

I've spent the last few days experimenting in building rust from a venv with no success, I've applied the patch from this PR on the current master and it worked on the first try 🎉

jcfr · 2018-08-10T18:09:38Z

Great 👍 Could you review this topic then: ~~master...jcfr:windows-copy-import-library-rebased ? A sanity check to make sure I properly rebased it.~~ EDIT: This topic has been updated.

macisamuele · 2018-08-10T18:24:22Z

@jcfr I can confirm. The build with your new branch worked as expected

jcfr · 2018-08-10T18:28:28Z

Thanks @macisamuele , topic updated 😄

gaborbernat

This will require a rebase on master, and some test to validate the changes. Thanks!

gaborbernat · 2019-01-14T13:36:20Z

Closing due to inactivity.

jcfr · 2019-01-14T13:59:58Z

@gaborbernat I would appreciate if you could give this a little more time. Thanks for your patience.

jcfr · 2019-01-14T21:51:28Z

Topic has been rebased and tested locally.

I confirm that the pythonXY.lib file is copied for all version of python:

$ find . | grep "\.lib$"
./venv-test-27-x64/libs/python27.lib
./venv-test-27-x86/libs/python27.lib
./venv-test-34-x64/libs/python34.lib
./venv-test-34-x86/libs/python34.lib
./venv-test-35-x64/libs/python35.lib
./venv-test-35-x86/libs/python35.lib
./venv-test-36-x64/libs/python36.lib
./venv-test-36-x86/libs/python36.lib
./venv-test-37-x64/libs/python37.lib
./venv-test-37-x86/libs/python37.lib

@gaborbernat I was thinking to update the test test_install_python_bin ? What do you think ?

…stutils This issue was discovered while using virtualenv to create an isolated environment and test that a source distribution would compile without using the knowledge hardcoded in distutils.sysconfig and virtualenv/virtualenv_embedded/distutils-init.py In other word, without relying on the fact virtualenv monkey patch distutils to ensure the value associated with LIBDIR variable is correct. This is relevant because the library location is implicitly specified in PC/pyconfig.h 8<---8<---8<---8<---8<---8<---8<---8<---8<---8<---8<---8<---8<---8<--- [...] /* For an MSVC DLL, we can nominate the .lib files used by extensions */ #ifdef MS_COREDLL # ifndef Py_BUILD_CORE /* not building the core - must be an ext */ # if defined(_MSC_VER) /* So MSVC users need not specify the .lib file in their Makefile (other compilers are generally taken care of by distutils.) */ # if defined(_DEBUG) # pragma comment(lib,"python35_d.lib") # elif defined(Py_LIMITED_API) # pragma comment(lib,"python3.lib") # else # pragma comment(lib,"python35.lib") # endif /* _DEBUG */ # endif /* _MSC_VER */ # endif /* Py_BUILD_CORE */ #endif /* MS_COREDLL */ [...] 8<---8<---8<---8<---8<---8<---8<---8<---8<---8<---8<---8<---8<---8<--- References: * https://docs.python.org/2/extending/windows.html#using-dlls-in-practice * https://docs.python.org/3.6/extending/windows.html#using-dlls-in-practice Tested-by: Samuele Maci <macisamuele@gmail.com>

gaborbernat

As far as modifying that test to assert against it, I'm good with it.

gaborbernat · 2019-01-15T10:37:51Z

virtualenv.py

+                # considering that on windows, python import libraries are located in
+                # the  "<root>/libs" directory, the following code will look for and
+                # copy "pythonXY.lib".
+                pythonlib_name = "python{}{}.lib".format(sys.version_info[0], sys.version_info[1])


slight name tweak, python_lib_name (the logic applies to all other names too)

gaborbernat · 2019-01-15T10:38:39Z

virtualenv.py

+                pythonlib_dest_dir = os.path.join(lib_dir, "..", "libs")
+                pythonlib_dest = os.path.join(pythonlib_dest_dir, pythonlib_name)
+                if os.path.exists(pythonlib):
+                    logger.info("Also created %s" % pythonlib_name)


Do not materialize the string yet, leave it extra arg to logger.

gaborbernat · 2019-01-15T10:39:45Z

virtualenv.py

@@ -1395,6 +1395,20 @@ def install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages, clear, sy
                elif os.path.exists(python_dll_d_dest):
                    logger.info("Removed %s as the source does not exist", python_dll_d_dest)
                    os.unlink(python_dll_d_dest)
+
+            if not IS_PYPY:


Why just pypy, what about jython? We should also mention why this is so special

Jython is java-based and doesn't provide import libraries like CPython does.

Pypy indeed provides a libs folder with the corresponding pythonXY.lib file, I will update the patch to not exclude it.

gaborbernat · 2019-01-15T10:40:19Z

virtualenv.py

+                # considering that on windows, python import libraries are located in
+                # the  "<root>/libs" directory, the following code will look for and
+                # copy "pythonXY.lib".
+                pythonlib_name = "python{}{}.lib".format(sys.version_info[0], sys.version_info[1])


is it lib on Windows too?

.lib is indeed the extension on Windows. Test above was performed on windows and the directory were searched running find and grep from Git Bash.

gaborbernat · 2019-01-15T10:41:52Z

virtualenv.py

+                # copy "pythonXY.lib".
+                pythonlib_name = "python{}{}.lib".format(sys.version_info[0], sys.version_info[1])
+                pythonlib = os.path.join(os.path.dirname(sys.executable), "libs", pythonlib_name)
+                pythonlib_dest_dir = os.path.join(lib_dir, "..", "libs")


Shouldn't this take the lib_dir value from

virtualenv/virtualenv.py

Line 1095 in 508919a

lib_dir = join(home_dir, "Lib")

?

The .lib file indeed lives in the .libs folder.

stale · 2019-04-15T20:40:52Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Just add a comment if you want to keep it open. Thank you for your contributions.

gaborbernat · 2019-11-16T17:57:13Z

Closing due to no action on the PR for a long time, if anyone has the bandwidth to pick this up, let me know, and we can have a talk what would be a good solution here.

jcfr · 2019-11-18T14:33:56Z

Hi @gaborbernat ,

Thanks for the update.

If I find some time, I will rebase, test locally and answer follow-up questions.

Best,

gaborbernat · 2019-11-18T14:36:18Z

I'd like to validate this on the new rewrite branch to be honest, though not sure is advanced enough to add this.

gaborbernat · 2019-11-18T14:46:02Z

created #1448 to track this

jcfr mentioned this pull request Feb 23, 2017

Copy libs/pythonXY.lib to support building of C extensions #992

Closed

jcfr force-pushed the windows-copy-import-library branch from c96b157 to 895742d Compare August 10, 2018 18:27

macisamuele mentioned this pull request Sep 8, 2018

Can't build pyo3 projects on Python 2.7 / Windows 10 PyO3/pyo3#215

Closed

gaborbernat force-pushed the master branch 6 times, most recently from 2b3f876 to 9dfcb43 Compare October 27, 2018 11:31

gaborbernat requested changes Dec 16, 2018

View reviewed changes

gaborbernat closed this Jan 14, 2019

gaborbernat reopened this Jan 14, 2019

jcfr force-pushed the windows-copy-import-library branch 3 times, most recently from 9354332 to 121cc1e Compare January 14, 2019 21:45

jcfr force-pushed the windows-copy-import-library branch from 121cc1e to 508919a Compare January 14, 2019 21:52

gaborbernat requested changes Jan 15, 2019

View reviewed changes

stale bot added the wontfix label Apr 15, 2019

gaborbernat removed the wontfix label Apr 16, 2019

gaborbernat closed this Nov 16, 2019

gaborbernat mentioned this pull request Nov 18, 2019

support building of C extensions - rewrite #1448

Closed

mattip mentioned this pull request Oct 4, 2021

Support PyPy 3.8 #2206

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Copy libs/pythonXY.lib to support building of C extensions without distutils #1023

Copy libs/pythonXY.lib to support building of C extensions without distutils #1023

jcfr commented Feb 23, 2017 •

edited

Loading

macisamuele commented Aug 10, 2018

jcfr commented Aug 10, 2018 •

edited

Loading

macisamuele commented Aug 10, 2018

jcfr commented Aug 10, 2018 •

edited

Loading

macisamuele commented Aug 10, 2018

jcfr commented Aug 10, 2018

gaborbernat left a comment

gaborbernat commented Jan 14, 2019

jcfr commented Jan 14, 2019

jcfr commented Jan 14, 2019

gaborbernat left a comment

gaborbernat Jan 15, 2019

gaborbernat Jan 15, 2019

gaborbernat Jan 15, 2019

jcfr Jan 15, 2019

gaborbernat Jan 15, 2019

jcfr Jan 15, 2019

gaborbernat Jan 15, 2019

jcfr Jan 15, 2019

stale bot commented Apr 15, 2019

gaborbernat commented Nov 16, 2019

jcfr commented Nov 18, 2019 •

edited

Loading

gaborbernat commented Nov 18, 2019

gaborbernat commented Nov 18, 2019

Copy libs/pythonXY.lib to support building of C extensions without distutils #1023

Copy libs/pythonXY.lib to support building of C extensions without distutils #1023

Conversation

jcfr commented Feb 23, 2017 • edited Loading

macisamuele commented Aug 10, 2018

jcfr commented Aug 10, 2018 • edited Loading

macisamuele commented Aug 10, 2018

jcfr commented Aug 10, 2018 • edited Loading

macisamuele commented Aug 10, 2018

jcfr commented Aug 10, 2018

gaborbernat left a comment

Choose a reason for hiding this comment

gaborbernat commented Jan 14, 2019

jcfr commented Jan 14, 2019

jcfr commented Jan 14, 2019

gaborbernat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stale bot commented Apr 15, 2019

gaborbernat commented Nov 16, 2019

jcfr commented Nov 18, 2019 • edited Loading

gaborbernat commented Nov 18, 2019

gaborbernat commented Nov 18, 2019

jcfr commented Feb 23, 2017 •

edited

Loading

jcfr commented Aug 10, 2018 •

edited

Loading

jcfr commented Aug 10, 2018 •

edited

Loading

jcfr commented Nov 18, 2019 •

edited

Loading