Python 3.9 wheel

[thetonus]

Describe your feature request

Some CI systems have base images for testing. Using something like python:latest would grab Python 3.9. Since there are no Python 3.9 wheels for Ray, there will be many silent and hard to debug failures in CI systems (speaking from experience here). Also, 3.9 is also the new stable version of Python, and Ray should be supporting that as well.

There may have to be a discussion amongst the Ray team how far many older versions they want to build wheels for. It would be nice if the Ray team could support 0.8.4, since that is what my team uses.

[wuisawesome]

Thanks for the request.

I think building python 3.9 wheels for future releases should be pretty uncontroversial.

I'm not as sure about backporting the builds to older releases though.

@barakmich @rkooo567 do you have any thoughts?

[rkooo567]

Yeah I don't think we will push old wheels for python 3.9. (It is also probably not working because it might not be compatible to python 3.9. For example, we needed some work to push python 3.8 wheels). We can probably try build wheels for python 3.9 in the next release, but depending on how hard it is, it can take more time!

[wumpus]

I'm attempting to use the released pypi ray on travisci with 3.9 and travis says:

ERROR: Could not find a version that satisfies the requirement ray (from paramsurvey==0.4.8.dev7+g42db8c6) (from versions: none)
ERROR: No matching distribution found for ray (from paramsurvey==0.4.8.dev7+g42db8c6)

Am I doing it wrong?

3.9 was released October 5, that's nearly a month ago. I feel bad that I didn't try to mark my python packages as supporting 3.9 until now. Do I need to wait for some future release of ray that supports 3.9?

[wuisawesome]

Yeah we don't have an official wheel for 3.9 yet (in fact I haven't had the time to test Ray on 3.9 at all yet).

Would you be interested in testing/contributing one?

[wumpus]

happy to test, with a clue I could probably build it.

[wuisawesome]

@wumpus I was just able to build Ray by adding (3, 9) to the supported version in python/setup.py then running pushd; pip install -e .; popd.

I'm going to start running unit tests and posting results here. Let me know if you're able to help out still.

I'll also post some additional results here in a bit.

[wumpus]

Sure, I'd love to help test.

[acxz]

I made a PR to get started on this work. It's pretty naive as I just based it off the time we dropped Python 3.5 support, but have it in the CI should help make the debugging process open.

Building works fine on my local machine.

[wuisawesome]

Thanks! Please let us know if you find the root cause of the segfaults! Also feel free to reach out if you have any questions.

[wumpus]

Is there a way for me to download and test your actual wheel?

I tried testing by doing the install myself with pip install -e git+https://github.com/acxz/ray.git@py39#egg=ray&subdirectory=python -- pip understands that syntax, but even though I installed cython first, the build is dying in numpy (which has no wheel for 3.9 and the older numpy version that ray insists on installing) for a call to tp_print, which is an obsolete cython routine.

Full output at https://dev.azure.com/lindahl0577/971ed932-5696-4977-a9e8-1362510cb7c3/_apis/build/builds/65/logs/26

[acxz]

Is there a way for me to download and test your actual wheel?

Not sure.

[wuisawesome]

the older numpy version that ray insists on installing

@wumpus

What version of numpy does it look like Ray is trying to install? I think Ray should only depend on numpy >= 1.16 (and even that is really a soft dependency for zero copy deserialization).

[wumpus]

The logfile I linked above says:

2020-12-04T05:46:18.3554527Z command: /opt/hostedtoolcache/Python/3.9.0/x64/bin/python /opt/hostedtoolcache/Python/3.9.0/x64/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-6udm73zk/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel

So it's installing numpy==1.16.0, which is old and does not have a 3.9 wheel.

[wuisawesome]

Ok, it sounds like we'll have to add a new rule like numpy >=1.19.0; python >= 3.9.

For now, can you try to manually upgrade numpy yourself (pip install --upgrade numpy and verify you have 1.19.x)?

[wumpus]

I'm using CI so I have to script the entire thing -- I can't figure out where in the ray repo where it is choosing numpy==1.16.0.

[wuisawesome]

Should be this: https://github.com/ray-project/ray/blob/master/python/setup.py#L142

[richardliaw]

Seems like more and more users are requesting this :)

ray-project/tune-sklearn#169

[acxz]

At this point we do have both linux and mac wheels building just fine, only waiting on windows support.

[wuisawesome]

At this point we do have both linux and mac wheels building just fine, only waiting on windows support.

Though to be clear, we haven't run unit tests against the wheels right?

[acxz]

Doesn't the CI handle that?

Or it runs the unit tests on the source and builds the wheels, but it doesn't run the unit tests after installing the built wheels from the CI runs?

[wuisawesome]

The CI only runs test_basic against the wheels as a sanity check, since running all the tests every time is super expensive/takes a long time.

https://github.com/ray-project/ray/blob/b56db5a2/ci/travis/test-wheels.sh#L24

[acxz]

Ah so then how do we unit test the wheels before release then? Do you just download/install the wheels from the CI and run the unit tests on your local machine? If so do you want to give that a shot @wuisawesome ?

[wuisawesome]

Yeah, since we're going to need to call it experimental for now, I think it's sufficient to just manually test the wheels to have some degree of confidence that it is behaving the same on different python versions.

[acxz]

@wuisawesome did you get a chance to test the wheels?

[wuisawesome]

I haven't had a chance to test your branch yet, though I just copied your PR into a ray-project/ray branch, so the CI will build some authentic wheels we can use.

Edit: I haven't had a chance to play around with it yet, but here' the linux wheel: https://ray-wheels.s3-us-west-2.amazonaws.com/python3.9/a902f2e4ab0a9c27ece8562084aa3fc4be68eeb8/ray-1.2.0.dev0-cp39-cp39-manylinux2014_x86_64.whl

[daniel347x]

Please forgive me if this is inappropriate to ask, as I am grateful for the effort folks have put into the Ray library, but is there a timeline for this? The lack of support by Ray for Python 3.9 is the only thing holding me back from upgrading to Python 3.9 at this point, and there is no good substitute for Ray as an improvement upon Python.Multiprocessing.

[satishdash]

Hi thanks a ton for all the efforts put in by the community for this awesome framework core and notably all the other components around it in the ecosystem. But by any chance do we know when would a compatible 3.9 wheel be built out. Eagerly awaited in our organization :)

[tbabej]

Since #12613 was reverted in #15003, should this issue be re-opened?

[wuisawesome]

Yes, hopefully #15006 will let us turn on support for windows and linux first at least.

[crazy25000]

Any update on this?

[tbabej]

Seconding this, having Python 3.9 wheel would be great. Major distributions are shipping Python 3.9 as their default Python version now (i.e. Ubuntu 21.04, Fedora 33/34).

[wuisawesome]

Yeah, I've been focusing most of my time on the 1.3 release, since that's been delayed. If someone can fix up #15006, that would be great, otherwise I'll try to work on it.

[tbabej]

@wuisawesome Understood. Getting Py3.9 wheels as part of the 1.3 release #15021 would be awesome.

[thomas-ward27]

In case this is any help to others;

We have just compiled ray from git master by merely adding 3,9 to SUPPORTED_PYTHONS in python/setup.py , we didn't include the CLI or dashboard.

So far ray works for our (very simple) use cases - we only use ray.init() one one machine and @ray.remote

Maybe it would be OK to add python 3.9 support to ray-master?

Below is what we did to compile on Debian bullseye (but running kernel 4.19 rather than the default 5 series)

git clone https://github.com/ray-project/ray.git

sudo apt-get install -y build-essential curl unzip psmisc

sudo pip3 install cython  pytest
sudo ray/ci/travis/install-bazel.sh

cd ray/python
mv setup.py setup.py.old
awk ' $1=="SUPPORTED_PYTHONS" { print "SUPPORTED_PYTHONS = [(3, 6), (3, 7), (3, 8), (3,9)]" ; getline } { print} ' setup.py.old >   setup.py
sudo pip3 install -e . --verbose

[tbabej]

I can confirm we built our own Python 3.9 wheels using the following patch on top of the ray-1.3.0 tag:

From 9c17c72fb633d40e2a720dd3a54fcc142166a27f Mon Sep 17 00:00:00 2001
From: Tomas Babej <tomas@tbabej.com>
Date: Sat, 8 May 2021 14:57:25 -0400
Subject: [PATCH] Support Python 3.9

---
 python/build-wheel-manylinux2014.sh | 8 ++------
 python/setup.py                     | 2 +-
 2 files changed, 3 insertions(+), 7 deletions(-)

diff --git a/python/build-wheel-manylinux2014.sh b/python/build-wheel-manylinux2014.sh
index 490ec256e..e30e8c2f3 100755
--- a/python/build-wheel-manylinux2014.sh
+++ b/python/build-wheel-manylinux2014.sh
@@ -12,15 +12,11 @@ EOF
 chmod +x /usr/bin/nproc
 
 NODE_VERSION="14"
-PYTHONS=("cp36-cp36m"
-         "cp37-cp37m"
-         "cp38-cp38")
+PYTHONS=("cp39-cp39")
 
 # The minimum supported numpy version is 1.14, see
 # https://issues.apache.org/jira/browse/ARROW-3141
-NUMPY_VERSIONS=("1.14.5"
-                "1.14.5"
-                "1.14.5")
+NUMPY_VERSIONS=("1.16.6")
 
 yum -y install unzip zip sudo
 yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel xz
diff --git a/python/setup.py b/python/setup.py
index 0575ad3a1..fdbabe006 100644
--- a/python/setup.py
+++ b/python/setup.py
@@ -27,7 +27,7 @@ logger = logging.getLogger(__name__)
 # before these files have been created, so we have to move the files
 # manually.
 
-SUPPORTED_PYTHONS = [(3, 6), (3, 7), (3, 8)]
+SUPPORTED_PYTHONS = [(3, 6), (3, 7), (3, 8), (3, 9)]
 SUPPORTED_BAZEL = (3, 2, 0)
 
 ROOT_DIR = os.path.dirname(__file__)
-- 
2.31.1

Note this builds Python 3.9 wheels only. To kick off the build process, use

docker run -e TRAVIS_COMMIT=<insert commit sha> --rm -w /ray -v `pwd`:/ray -ti quay.io/pypa/manylinux2014_x86_64  /ray/python/build-wheel-manylinux2014.sh

[cread]

Any progress on this? I was hoping that with the release of Ray 1.4.0 (congrats 🎉 ) this would all be sorted out, but sadly that's not the case.

[mwtian]

Ray 1.4.1 is available now, which provides Python 3.9 wheels.