build a single wheel for cpython 3 using py_limited_api #32

anthrotype · 2018-01-06T17:12:31Z

wheel 0.30.0 added a new --py-limited-api {cp32|cp33|cp34|...} option that produces cpNN.abi3.{arch} tags on CPython 3, for distributing wheels containing extension modules which define the Py_LIMITED_API

CFFI does that by default since v1.8 (see cffi docs)

This means CFFI extension modules are now named *.abi3.so, and should work on any version of CPython >= 3.2 (that's when the stabe Py_LIMITED_API was first introduced).

In setup.cfg, I still kept the minimum version supported to 3.4.

This new bdist_wheel option should be ignored on implementations different from cpython.

codecov · 2018-01-06T17:41:58Z

Codecov Report

Merging #32 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           master    #32   +/-   ##
=====================================
  Coverage     100%   100%           
=====================================
  Files           7      7           
  Lines         151    142    -9     
  Branches       15     15           
=====================================
- Hits          151    142    -9

Impacted Files	Coverage Δ
src/argon2/__init__.py	`100% <0%> (ø)`	⬆️
src/argon2/low_level.py	`100% <0%> (ø)`	⬆️
src/argon2/_password_hasher.py	`100% <0%> (ø)`	⬆️
src/argon2/exceptions.py	`100% <0%> (ø)`	⬆️
src/argon2/__main__.py	`100% <0%> (ø)`	⬆️
src/argon2/_legacy.py	`100% <0%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 84aeae0...0e8efb8. Read the comment docs.

anthrotype · 2018-01-06T17:55:15Z

Hm.. I have a feeling this bdist_wheel --py-limited-api option won't work on Windows, as different cpython minor versions may link with different versions of the MSVC runtime.

This is the error Appveyor is giving, even after ensuring wheel is up-to-date:

AssertionError: would build wheel with unsupported tag ('cp34', 'abi3', 'win32')

https://ci.appveyor.com/project/hynek/argon2-cffi/build/1.0.195/job/r9ank20duuyc2mun

@agronholm could you advise?

agronholm · 2018-01-06T19:15:48Z

Hm.. I have a feeling this bdist_wheel --py-limited-api option won't work on Windows, as different cpython minor versions may link with different versions of the MSVC runtime.

This is specifically one of the problems PEP 384 was intended to overcome – to avoid dependencies on specific C runtime versions.
I was wondering – do you even need the C code to access Python types? If not, have you considered using milksnake instead?

anthrotype · 2018-01-06T19:22:23Z

milnksnake is nice, but it's just a helper for CFFI in ABI mode (i.e. using shared library and dlopen), whereas this module uses cffi in API mode (i.e. set_source, etc.).
I still don't understand whether this new wheel option is intended to work on Windows as well as Unix.
I'll keep investigating.

agronholm · 2018-01-06T19:26:03Z

I will try to see if I can understand why that error message pops up.

anthrotype · 2018-01-06T19:30:59Z

maybe it has to do with this warning printed when running wheel.pep425tags.get_supported():

C:\Users\appveyor\AppData\Roaming\Python\Python34\site-packages\wheel\pep425tags.py:75: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
  warn=(impl == 'cp')):
C:\Users\appveyor\AppData\Roaming\Python\Python34\site-packages\wheel\pep425tags.py:79: RuntimeWarning: Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
  warn=(impl == 'cp')):

agronholm · 2018-01-06T19:34:11Z

imp.get_suffixes() does not seem to contain any ABI3 suffixes, but I'm not sure if this is even applicable on Windows. Frankly I'm out of my depth here, as I'm just the new maintainer and have not been involved in the design of this system.

anthrotype · 2018-01-06T19:36:35Z

no problem, thanks for checking! Shall we open a new issue over at pypa/wheel for this?

agronholm · 2018-01-06T19:37:21Z

I found this: https://mail.python.org/pipermail/distutils-sig/2013-February/020022.html

Please only open issues for solvable problems regarding the software, not the packaging standards!

anthrotype · 2018-01-06T19:42:07Z

PyQt5 provides abi3 wheels for manylinux and macosx and windows wheels with "none" abi tag: https://pypi.python.org/pypi/PyQt5/5.9.2

but they only support cpython 3.5 and above (which I think are built using the same Visual Studio 2015), whereas 3.4 uses 2010.

agronholm · 2018-01-06T19:45:44Z

none implies no C extensions (using the Python ABI). If they do use them, then the tag is wrong.

agronholm · 2018-01-06T19:47:16Z

If you want to get to the bottom of this, I suggest you open a new discussion thread on distutils-sig.

anthrotype · 2018-01-06T19:54:22Z

none implies no C extensions

well according to the thread you linked above, "none" is the only available ABI tag on Windows...

I suggest you open a new discussion thread on distutils-sig

I would like to get to the bottom of this, but I feel like this is a bit over my head.

agronholm · 2018-01-06T19:59:11Z

Well, it boils down to the question: Should ABI3 be usable on Windows? If so, what will it take?

agronholm · 2018-01-06T20:00:37Z

well according to the thread you linked above, "none" is the only available ABI tag on Windows...

No, Windows does support tags like cp34m as usual. It's just ABI3 that is the problem here.

anthrotype · 2018-01-07T12:41:07Z

I asked the question on distutils-sig.
I also found this related pip issue: pypa/pip#4445
I'm even more confused.

anthrotype · 2018-01-07T18:57:55Z

Slowly making progress. After ruling out an issue with setuptools, I found another issue, this time with cffi itself.

The latter's _cffi_include.h defines the Py_LIMITED_API too late, only after pyconfig.h has already been included. This means that the incorrect PYTHON36.DLL is linked with the extension, instead of the stable api's PYTHON3.DLL.

https://bitbucket.org/cffi/cffi/issues/350/issue-with-py_limited_api-on-windows

The only workaround so far is to pass define_macros=[("Py_LIMITED_API", None)] explicitly to the ffi.set_source call.

The same argon2._ffi extension module can then imported on Windows CPython 3.5 and 3.6, same as on manylinux1 and macosx. Cool! 😎

However, the issue with pep425tags and abi3 tag is still present on Windows: which means one can't bdist_wheel --py-limited-api cp35 or else one gets the AssertionError about the unsupported abi3 tag.

Interstingly, I can build the wheel on Windows without the --py-limited-api option, and then simply rename it to *-cp35.cp36.cp37-none-win_amd64 (note the none as the abi tag) and then I can successfully install and import the wheel in all the listed cpython versions.

However, inspecting the WHEEL metadata file in the zipped whl package revelas that the Tag is still cp36-cp36m-win_amd64 so we are kind of lying...

anthrotype · 2018-01-08T12:18:13Z

The issue with cffi and Py_LIMITED_API linking to the wrong python DLL on Windows has already been fixed on cffi master branch.

@hynek I hope you don't mind if I'm using your project as a guinea pig for testing the Py_LIMITED_API support in the current python packaging tools :)

hynek · 2018-01-08T16:56:02Z

just let me know when you need something ;)

anthrotype · 2018-01-13T10:00:16Z

Status update: cffi decided to revert the patch that fixed support for Py_LIMITED_API on windows.
The reason is that the current virtualenv does not correctly copy the required PYTHON3.DLL to the env’s Scripts directory and thus import fails.

https://bitbucket.org/cffi/cffi/issues/355/importerror-dll-load-failed-on-windows

The bug was fixed months ago on virtualenv master, but there hasn’t been a release since November 2016.

Pity.. I think we won’t be shipping abi3 wheels for another year.

anthrotype · 2018-01-17T10:10:45Z

just let me know when you need something ;)

@hynek maybe you know somebody at PyPA that could help moving this issue forward?
I wrote several times to distutils-sig in the last couple of weeks but didn't manage to get their attention (perhaps i'm too verbose).
This is the latest one: https://mail.python.org/pipermail/distutils-sig/2018-January/031856.html

Besides the issue with virtualenv not copying PYTHON3.DLL (easily fixed by tagging a new release, see pypa/virtualenv#1122), the other remaining issue is bdist_wheel not recognizing "abi3" as a valid tag on windows. It's not clear whether this is a limitation of the current packaging tools or rather an unsolved problem in the relevant PEPs (as @agronholm suggested above).

Maybe if @hynek steps in, they will listen to him.

pypa/wheel@28b735a https://github.com/pypa/wheel/blob/master/CHANGES.txt

…above wheel 0.30.0 added a new --py-limited-api {cp32|cp33|cp34|...} option that produces cpNN.abi3.{arch} tags on CPython 3, for distributing wheels containing extension modules which define the Py_LIMITED_API: https://docs.python.org/3/c-api/stable.html CFFI does that by default since v1.8: http://cffi.readthedocs.io/en/latest/cdef.html#ffibuilder-compile-etc-compiling-out-of-line-modules This means CFFI extension modules are now named *.abi3.so, and should work on any version of CPython >= 3.2 (that's when the stabe Py_LIMITED_API was first introduced). In setup.cfg, I still kept the minimum version supported to 3.4.

Maybe it will fix this error: https://ci.appveyor.com/project/hynek/argon2-cffi/build/1.0.194/job/ih6bg74p2h6ay5sc#L357

hynek · 2018-07-17T15:18:48Z

not giving up huh? :D

(I got a notification from AppVeyor)

anthrotype · 2018-07-17T15:31:00Z

I saw some activity on the pypa/wheel repo and wanted to see if anything had changed since last time.. but apparently no.
It's a pity nobody seems to care about this issue with Windows and stable ABI tag... 😢

agronholm · 2018-07-17T15:39:54Z

As wheel maintainer I will do whatever I can to fix this, but I don't know what to do! I posted a message on distutils-sig regarding the state of wheel.

anthrotype · 2018-07-17T15:46:03Z

@agronholm thanks, I really appreciate that!

hynek · 2019-10-27T19:00:56Z

I believe we ship abi3/limited_api wheels now using https://github.com/hynek/argon2-cffi/blob/master/.azure-pipelines/wheel-builder.yml – is that what you meant?

hynek · 2019-10-30T06:00:59Z

I’m taking the silence as confirmation ;)

anthrotype · 2019-10-30T12:47:34Z

Thanks, I'm glad that that at least for macos and linux we could build abi3 wheels!
I haven't checked this in a while on the status of py_limited_api for Windows (mostly gave up on it). Do you know if there was any progress on that front?

Also I wonder, why are you pinning an older version of pip in your wheel-builder.yml?

hynek · 2019-10-30T13:04:47Z

As far as I know, there's no abi3 wheels for Windows.

We need to pin pip because newer version support pep517 but no passing arguments (= abi3) to wheel which makes it explode. This is terrible, but I don't know what to do about it since the pip project is informed since January or February.

agronholm · 2019-10-30T13:12:09Z

So why do you need to pass it as an argument? This thread mentions setting this option in setup.cfg. What am I missing?

hynek · 2019-10-30T13:13:57Z

It's about pypa/pip#6304 – according to Paul and Alex there's no other way? I'm more than happy to be corrected.

agronholm · 2019-10-30T13:17:10Z

To the best of my understanding, the pep517 command should invoke the setuptools build backend which then should invoke wheel, which does read this option in setup.cfg. I have not tested this locally yet, and I could be mistaken about how the processing chain works.

hynek · 2019-10-31T04:33:25Z

My understanding of the problem here is that there are no abi3 for Windows, so we need to set the flags on a case to case basis. :(

anthrotype mentioned this pull request Jan 7, 2018

extension with py_limited_api=True not linked with PYTHON3.DLL pypa/setuptools#1248

Closed

anthrotype mentioned this pull request Mar 31, 2018

Windows Stable ABI Support pypa/pip#4445

Closed

hynek force-pushed the master branch from e67d4ac to fe591a6 Compare June 16, 2018 09:30

anthrotype added 3 commits July 17, 2018 16:51

setup.py: require wheel >= 0.30.0 with support for py_limited_api

68bb8e4

pypa/wheel@28b735a https://github.com/pypa/wheel/blob/master/CHANGES.txt

appveyor: upgrade wheel package as well

0e8efb8

Maybe it will fix this error: https://ci.appveyor.com/project/hynek/argon2-cffi/build/1.0.194/job/ih6bg74p2h6ay5sc#L357

anthrotype force-pushed the abi3 branch from fc9775b to 0e8efb8 Compare July 17, 2018 14:52

hynek closed this Oct 30, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build a single wheel for cpython 3 using py_limited_api #32

build a single wheel for cpython 3 using py_limited_api #32

anthrotype commented Jan 6, 2018 •

edited

Loading

codecov bot commented Jan 6, 2018 •

edited

Loading

anthrotype commented Jan 6, 2018

agronholm commented Jan 6, 2018

anthrotype commented Jan 6, 2018

agronholm commented Jan 6, 2018

anthrotype commented Jan 6, 2018

agronholm commented Jan 6, 2018

anthrotype commented Jan 6, 2018

agronholm commented Jan 6, 2018

anthrotype commented Jan 6, 2018

agronholm commented Jan 6, 2018 •

edited

Loading

agronholm commented Jan 6, 2018

anthrotype commented Jan 6, 2018

agronholm commented Jan 6, 2018

agronholm commented Jan 6, 2018 •

edited

Loading

anthrotype commented Jan 7, 2018

anthrotype commented Jan 7, 2018 •

edited

Loading

anthrotype commented Jan 8, 2018

hynek commented Jan 8, 2018

anthrotype commented Jan 13, 2018 •

edited

Loading

anthrotype commented Jan 17, 2018

hynek commented Jul 17, 2018

anthrotype commented Jul 17, 2018

agronholm commented Jul 17, 2018

anthrotype commented Jul 17, 2018

hynek commented Oct 27, 2019

hynek commented Oct 30, 2019

anthrotype commented Oct 30, 2019

hynek commented Oct 30, 2019

agronholm commented Oct 30, 2019

hynek commented Oct 30, 2019

agronholm commented Oct 30, 2019

hynek commented Oct 31, 2019

build a single wheel for cpython 3 using py_limited_api #32

build a single wheel for cpython 3 using py_limited_api #32

Conversation

anthrotype commented Jan 6, 2018 • edited Loading

codecov bot commented Jan 6, 2018 • edited Loading

Codecov Report

anthrotype commented Jan 6, 2018

agronholm commented Jan 6, 2018

anthrotype commented Jan 6, 2018

agronholm commented Jan 6, 2018

anthrotype commented Jan 6, 2018

agronholm commented Jan 6, 2018

anthrotype commented Jan 6, 2018

agronholm commented Jan 6, 2018

anthrotype commented Jan 6, 2018

agronholm commented Jan 6, 2018 • edited Loading

agronholm commented Jan 6, 2018

anthrotype commented Jan 6, 2018

agronholm commented Jan 6, 2018

agronholm commented Jan 6, 2018 • edited Loading

anthrotype commented Jan 7, 2018

anthrotype commented Jan 7, 2018 • edited Loading

anthrotype commented Jan 8, 2018

hynek commented Jan 8, 2018

anthrotype commented Jan 13, 2018 • edited Loading

anthrotype commented Jan 17, 2018

hynek commented Jul 17, 2018

anthrotype commented Jul 17, 2018

agronholm commented Jul 17, 2018

anthrotype commented Jul 17, 2018

hynek commented Oct 27, 2019

hynek commented Oct 30, 2019

anthrotype commented Oct 30, 2019

hynek commented Oct 30, 2019

agronholm commented Oct 30, 2019

hynek commented Oct 30, 2019

agronholm commented Oct 30, 2019

hynek commented Oct 31, 2019

anthrotype commented Jan 6, 2018 •

edited

Loading

codecov bot commented Jan 6, 2018 •

edited

Loading

agronholm commented Jan 6, 2018 •

edited

Loading

agronholm commented Jan 6, 2018 •

edited

Loading

anthrotype commented Jan 7, 2018 •

edited

Loading

anthrotype commented Jan 13, 2018 •

edited

Loading