ARROW-6920: [Packaging] Build python 3.8 wheels #5685

sjhewitt · 2019-10-17T17:15:55Z

adds python3.8 wheels

as far as I can tell python3.8 isn't available for Conda yet (conda-forge/python-feedstock#274), so that's will have to be added later

github-actions · 2019-10-17T17:17:56Z

https://issues.apache.org/jira/browse/ARROW-6920

kszucs · 2019-10-17T17:46:42Z

Thanks @sjhewitt! I'm submitting the tasks to see whether they are working.

kszucs · 2019-10-17T17:47:32Z

@ursabot crossbow submit wheel-manylinux1-cp38m wheel-manylinux2010-cp38m wheel-win-cp38m wheel-osx-cp38m

ursabot · 2019-10-17T17:47:52Z

AMD64 Conda Crossbow Submit (#71295) builder has been succeeded.

Revision: 9e6566c

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-269

Task	Status
wheel-manylinux1-cp38m
wheel-manylinux2010-cp38m
wheel-osx-cp38m
wheel-win-cp38m

kszucs · 2019-10-17T18:01:39Z

Well, we'll need to add a couple more things:

on windows we use conda forge to build the wheels, so I'm afraid we need to wait until they ship python 3.8
the manylinux containers don't contain the 3.8 interpreters so we need to add it and rebuild both manylinux1 and manylinux2010
osx wheels dependent on ARROW-6937: [Packaging][Python] Fix conda linux and OSX wheel nightly builds #5647

sjhewitt · 2019-10-17T18:52:16Z

@kszucs it looks like 3.8 was added to the base image that the manylinux builds from a few days ago: pypa/manylinux#370

kszucs · 2019-10-18T18:00:59Z

Yes, I'll rebuild the images.

sjhewitt · 2019-10-22T16:42:33Z

@kszucs any luck rebuilding the base images?

kszucs · 2019-10-22T21:40:20Z

Haven't had the time yet, but I'll try to build them tomorrow.

wesm · 2019-11-04T19:39:53Z

@kszucs @sjhewitt should we try to pick this one back up? We can probably try to upload unofficial/unsigned 3.8 wheels for 0.15.1

pitrou · 2019-11-06T09:52:49Z

@ursabot crossbow submit wheel-manylinux1-cp38m wheel-manylinux2010-cp38m wheel-win-cp38m wheel-osx-cp38m

ursabot · 2019-11-06T09:53:11Z

AMD64 Conda Crossbow Submit (#75525) builder has been succeeded.

Revision: 9e6566c

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-306

Task	Status
wheel-manylinux1-cp38m
wheel-manylinux2010-cp38m
wheel-osx-cp38m
wheel-win-cp38m

thehesiod · 2019-11-12T19:27:36Z

I've been trying to get this to work via the docker-compose.yml file, so far the build_virtualenv.sh file required the most changes, mostly bumping cython, etc versions to versions that have 3.8 wheels and the path to 3.8 is wrong as it isn't 38m

thehesiod · 2019-11-12T20:48:07Z

yay, got it to build. the cpython_path thing is wrong for 3.8

thehesiod · 2019-11-12T22:41:37Z

ugh, now I ran into the fact that the manylinux2010 links against libcrypt.so.2 which is not available on debian:buster

thehesiod · 2019-11-12T23:12:04Z

looks like the 38 cpython path issues are fixed in a newer sha of https://github.com/matthew-brett/multibuild/blob/devel/manylinux_utils.sh so this line: it clone https://github.com/matthew-brett/multibuild.git && cd multibuild && git checkout 1a7f31be677185f2dface2643284846e14130c3f needs to be updated

terencehonles · 2019-11-21T17:49:40Z

Any updates to this? We're looking forward to using Python 3.8, but this is a blocker for us and we don't want to manually build ourselves if this might be coming soon :)

kou · 2019-11-22T02:52:56Z

@ursabot crossbow submit wheel-manylinux1-cp38 wheel-manylinux2010-cp38 wheel-win-cp38 wheel-osx-cp38

ursabot · 2019-11-22T02:54:31Z

AMD64 Conda Crossbow Submit (#78543) builder has been succeeded.

Revision: 0edb002

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-336

Task	Status
wheel-manylinux1-cp38
wheel-manylinux2010-cp38
wheel-osx-cp38
wheel-win-cp38

kou · 2019-11-22T09:07:30Z

@ursabot crossbow submit wheel-osx-cp38

ursabot · 2019-11-22T09:07:54Z

AMD64 Conda Crossbow Submit (#78589) builder has been succeeded.

Revision: 717f705

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-337

Task	Status
wheel-osx-cp38

kou · 2019-11-22T09:14:15Z

I tried this.

Linux:

I could build manylinux1 and manylinux2010 wheels on local machine.
Travis CI build is broken because of OpenSSL related problem. This is not related to this change.

Windows:

Wheel can be built on AppVeyor: https://ci.appveyor.com/project/Ursa-Labs/crossbow/builds/29040257?fullLog=true#L4300
Can't be tested because msgpack-python doesn't provide wheel for Python 3.8 yet: https://ci.appveyor.com/project/Ursa-Labs/crossbow/builds/29040257?fullLog=true#L5460

macOS:

Testing: https://travis-ci.org/ursa-labs/crossbow/builds/615455584

kou · 2019-11-22T22:49:03Z

For Linux problem: #5886

terencehonles · 2019-11-22T22:53:28Z

@kou I added msgpack/msgpack-python#374 to get the wheels built.

kou · 2019-11-22T22:53:48Z

Great!

kou · 2019-11-24T00:22:38Z

For macOS problem: #5887

kou · 2019-11-24T02:04:54Z

@ursabot crossbow submit -g wheel

ursabot · 2019-11-24T02:07:41Z

AMD64 Conda Crossbow Submit (#78776) builder has been succeeded.

Revision: 1e34011

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-342

Task	Status
wheel-manylinux1-cp27m
wheel-manylinux1-cp27mu
wheel-manylinux1-cp35m
wheel-manylinux1-cp36m
wheel-manylinux1-cp37m
wheel-manylinux1-cp38
wheel-manylinux2010-cp27m
wheel-manylinux2010-cp27mu
wheel-manylinux2010-cp35m
wheel-manylinux2010-cp36m
wheel-manylinux2010-cp37m
wheel-manylinux2010-cp38
wheel-osx-cp27m
wheel-osx-cp35m
wheel-osx-cp36m
wheel-osx-cp37m
wheel-osx-cp38
wheel-win-cp36m
wheel-win-cp37m
wheel-win-cp38

terencehonles · 2019-12-07T20:08:14Z

@kou 5293f2b nice! I was going to ask why the venvs were not specified with a requirements file, and that's when you updated the numpy version in the wheel requirements.

I was also not sure if the tensorflow < 3.7 was still applicable. Looks like you got that too! 💪

kou · 2019-12-07T21:13:20Z

@ursabot crossbow submit wheel-manylinux2010-cp38

ursabot · 2019-12-07T21:13:47Z

AMD64 Conda Crossbow Submit (#81731) builder has been succeeded.

Revision: 908a396

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-390

Task	Status
wheel-manylinux2010-cp38

kszucs

@kou it looks good to me, feel free to merge once the builds are passing.

kou · 2019-12-07T21:55:59Z

OK!

kou · 2019-12-07T21:56:14Z

@ursabot crossbow submit -g wheel

ursabot · 2019-12-07T21:56:48Z

AMD64 Conda Crossbow Submit (#81748) builder has been succeeded.

Revision: feac494

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-391

Task	Status
wheel-manylinux1-cp27m
wheel-manylinux1-cp27mu
wheel-manylinux1-cp35m
wheel-manylinux1-cp36m
wheel-manylinux1-cp37m
wheel-manylinux1-cp38
wheel-manylinux2010-cp27m
wheel-manylinux2010-cp27mu
wheel-manylinux2010-cp35m
wheel-manylinux2010-cp36m
wheel-manylinux2010-cp37m
wheel-manylinux2010-cp38
wheel-osx-cp27m
wheel-osx-cp35m
wheel-osx-cp36m
wheel-osx-cp37m
wheel-osx-cp38
wheel-win-cp36m
wheel-win-cp37m
wheel-win-cp38

kou · 2019-12-07T22:01:53Z

@terencehonles Could you test some built wheels? You can find them at "ursabot-391-XXX" in https://github.com/ursa-labs/crossbow/releases .

I'll merged this once you confirm that built wheels work well. Thanks for your contribution!

terencehonles · 2019-12-08T21:17:35Z

@kou we don't have a particularly comprehensive test suite, but I confirmed the wheel installs, can output a feather file, and the file can be read back into pandas. I checked feather and it doesn't have any tests of its own.

I also ran the pyarrow tests in our container cd /tmp; pytest -r s --pyargs pyarrow -m 'not plasma and not requires_testing_data', and all tests expected to pass passed

kou · 2019-12-08T21:22:01Z

Thanks!
I'll merge this.

terencehonles · 2019-12-10T15:28:55Z

@kou do you know if/when the 0.15.1 Python 3.8 wheels are going to get published to pypi or will it be just for the next version? If the next version do you have any ideas on when that might be?

Thanks in advance!

wesm · 2019-12-10T15:33:26Z

It's probably not going to be until the next version, unless someone wants to produce 0.15.1 wheel files for us to upload to PyPI (if you want to volunteer to do this...be our guest)

FWIW Python 3.8 came out in October of this year -- with not even 2 months having passed, it feels extremely early to be considering 3.8.x for production use (personally I wouldn't consider until 6 months have elapsed). You can build and install your own wheels using the manylinux scripts in the meantime though

terencehonles · 2019-12-10T16:04:29Z

@wesm this PR generated 3.8 wheels, which is why I'm asking 😅. It's fine if it's next release that there's an upload to pypi, since I understand the wheels are built against master and I don't know what other changes are on master.

wesm · 2019-12-10T16:07:46Z

"this PR generated 3.8 wheels" -- not exactly. It generates 3.8 wheels on the master branch, which is not what is on PyPI. If you want to try to build wheels for the 0.15.1 tag by splicing in these changes you are welcome to try, but it is not going to be as simple as running one or two shell commands

terencehonles · 2019-12-10T16:27:07Z

Yes, I got that. I mentioned it was building against master. I can try cherry picking the commit, but you answered my question.

The other question is some insight into when a next release might happen. I'm not sure how useful trying to build the wheels will be if a release drops in the next couple weeks.

wesm · 2019-12-10T16:32:01Z

It's hard to predict because this is a big release with a bunch of things still up in the air. Best guess would be end of January but it may be sometime in February

terencehonles · 2019-12-10T17:37:06Z

Thanks @wesm

kou · 2019-12-10T21:42:22Z

FYI: We need to backport at least https://github.com/apache/arrow/pull/5685/files#diff-b10dcad1f78a3bf3bc5a5724ec057381 . It means that we can't generate wheel for Python 3.8 with 0.15.1.

terencehonles · 2019-12-11T02:59:27Z

Cool, thanks for the heads up @kou

AlJohri · 2019-12-26T16:04:11Z

I just ran into this as well where pip install pyarrow was failing on Python 3.8. Took a while to find the issue.

It would be great if pyarrow was explictly pinned to <3.8 or <3.9 in the future. In addition, I was confused as to why cmake wasn't telling me exactly what the error is (even in verbose mode) which made it much harder to google.

cmake -DPYTHON_EXECUTABLE=/Users/johria/Development/dem-primary-media-trait-analysis/.venv/bin/python3 -DPYARROW_BOOST_USE_SHARED=on -DCMAKE_BUILD_TYPE=release /private/var/folders/wc/5cw9h18x48z9h4dwfk1tk7n5jhnlj6/T/pip-install-s21y4ipy/pyarrow error: command 'cmake' failed with exit status 1
Building wheel for pyarrow (PEP 517) ... error
ERROR: Failed building wheel for pyarrow

kou · 2019-12-26T23:58:33Z

Could you open a new issue on JIRA? https://issues.apache.org/jira/projects/ARROW/issues/

I'm not sure whether pinning to "<3.8" is better or not but it's different topic of this.

@kou

…nd Windows wheel Adds support and testing for Python 3.9. I am looking for review as this change may have touched too many things, but I'm also looking to get the CI to test all the different environments. H/T: @kou, the documentation and #5685 for helping me get this off the ground. Closes #8386 from terencehonles/arrow-10224 Authored-by: Terence D. Honles <terence@honles.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

@kou

…nd Windows wheel Adds support and testing for Python 3.9. I am looking for review as this change may have touched too many things, but I'm also looking to get the CI to test all the different environments. H/T: @kou, the documentation and apache#5685 for helping me get this off the ground. Closes apache#8386 from terencehonles/arrow-10224 Authored-by: Terence D. Honles <terence@honles.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

kou force-pushed the arrow-6920 branch from 9e6566c to ab0623a Compare November 21, 2019 21:43

kou force-pushed the arrow-6920 branch from 717f705 to 1e34011 Compare November 24, 2019 02:04

kou added 2 commits December 8, 2019 06:12

Add Python 3.8 support to package metadata

43833a3

Check Docker version for debug

908a396

kszucs approved these changes Dec 7, 2019

View reviewed changes

Add a TODO comment

feac494

kou closed this in 6c6754a Dec 8, 2019

hugomailhot mentioned this pull request Jan 11, 2020

Failed to build pyarrow on python 3.8 #5987

Closed

terencehonles mentioned this pull request Oct 7, 2020

ARROW-10224: [Python] Add support for Python 3.9 except macOS wheel and Windows wheel #8386

Closed

asfimport mentioned this pull request Dec 26, 2019

[Python] create manylinux wheels for python3.8 #23243

Closed

ARROW-6920: [Packaging] Build python 3.8 wheels #5685

ARROW-6920: [Packaging] Build python 3.8 wheels #5685

Conversation

sjhewitt commented Oct 17, 2019

github-actions bot commented Oct 17, 2019

kszucs commented Oct 17, 2019

kszucs commented Oct 17, 2019

ursabot commented Oct 17, 2019

kszucs commented Oct 17, 2019 • edited Loading

sjhewitt commented Oct 17, 2019 • edited Loading

kszucs commented Oct 18, 2019

sjhewitt commented Oct 22, 2019

kszucs commented Oct 22, 2019

wesm commented Nov 4, 2019

pitrou commented Nov 6, 2019

ursabot commented Nov 6, 2019

thehesiod commented Nov 12, 2019

thehesiod commented Nov 12, 2019

thehesiod commented Nov 12, 2019

thehesiod commented Nov 12, 2019

terencehonles commented Nov 21, 2019

kou commented Nov 22, 2019

ursabot commented Nov 22, 2019

kou commented Nov 22, 2019

ursabot commented Nov 22, 2019

kou commented Nov 22, 2019

kou commented Nov 22, 2019

terencehonles commented Nov 22, 2019

kou commented Nov 22, 2019

kou commented Nov 24, 2019

kou commented Nov 24, 2019

ursabot commented Nov 24, 2019

terencehonles commented Dec 7, 2019

kou commented Dec 7, 2019

ursabot commented Dec 7, 2019

kszucs left a comment

Choose a reason for hiding this comment

kou commented Dec 7, 2019

kou commented Dec 7, 2019

ursabot commented Dec 7, 2019

kou commented Dec 7, 2019

terencehonles commented Dec 8, 2019

kou commented Dec 8, 2019

terencehonles commented Dec 10, 2019

wesm commented Dec 10, 2019

terencehonles commented Dec 10, 2019

wesm commented Dec 10, 2019

terencehonles commented Dec 10, 2019

wesm commented Dec 10, 2019

terencehonles commented Dec 10, 2019

kou commented Dec 10, 2019

terencehonles commented Dec 11, 2019

AlJohri commented Dec 26, 2019 • edited Loading

kou commented Dec 26, 2019

kszucs commented Oct 17, 2019 •

edited

Loading

sjhewitt commented Oct 17, 2019 •

edited

Loading

AlJohri commented Dec 26, 2019 •

edited

Loading