Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Default to Python 3.7 #10319

Merged
merged 2 commits into from
Oct 22, 2019
Merged

Default to Python 3.7 #10319

merged 2 commits into from
Oct 22, 2019

Conversation

adamjstewart
Copy link
Member

@adamjstewart adamjstewart commented Jan 11, 2019 •
edited
Loading

Closes #7926
Closes #8018
Closes #11468

Apologies for making a PR on such a controversial topic without first opening an issue for discussion, but I wanted to make it clear what I'm proposing. This PR should not be merged without majority approval from several people.

Currently, spack install python defaults to Python 2. With this PR, Spack will install Python 3 by default. This PR is motivated by the fact that Python 2 will be retired at the end of the year.

Currently, the following packages only support Python 2:

And the following packages only support Python 3:

* The latest version of these packages require Python 3, although older versions can be built with Python 2.

Feel free to update this list as it changes. Many of the packages that only support Python 2 may have already been updated upstream but the Spack package simply contains an old version. It would be good to push the developers of the rest of the packages to support Python 3.

Currently, there are more packages that require Python 2 than there are that require Python 3. But this will continue to change as the countdown clock reaches 2020. In particular, the reason I opened this PR is the meson package, which is becoming the default build system for GTK and many GNOME packages. Meson requires Python 3, and trying to build anything with meson anywhere in the spec causes Spack to crash. This could obviously be resolved with better concretization, but I haven't seen any progress made on that.

Some possibly biased Python 3 users who would be interested in this topic: @svenevs @citibeth @tgamblin

@tgamblin
Copy link
Member

I like this in principle, but I suspect that it will surprise a lot of users. I'm also not sure how robust the python 3 support is yet. Can we hold off until we can ensure that a decent subset of the python 3 packages build fine as binaries? I'm fine with having an opinion about Python as long as 3 works as well as 2, but right now I worry that we'll be enforcing this and providing a less robust default stack.

@citibeth
Copy link
Member

citibeth commented Jan 13, 2019 via email

@citibeth citibeth changed the title Default to Python 3 Default to Python3 Jan 15, 2019
@citibeth
Copy link
Member

@adamjstewart Have you reviewed / incorporated #7926 in this PR? Currently, that PR is required to build Python3-based software stacks. It could be merged in a way that allows both Python2 and Python3-based stacks. If we're going to have Python3 be the default, we might wish to reverse the "sense" of #7926 so that extra variants are required for Python2.

#7926 would not be needed with a better concretizer. But we don't currently have one.

@citibeth
Copy link
Member

citibeth commented Jan 15, 2019 •
edited
Loading

My thoughts on this:

Apart from Spack, the Python3 ecosystem is rather mature. Most things that most people want to do "just work." All the "core" Python libraries work fine with Python3. I've been running a Python3 stack with Spack for 3 years, and it works great.

Setting the default version of Python to 3.x in your packages.yaml file is not very hard. Therefore, this PR doesn't do much.

HOWEVER... the reality is that, as it stands, Spack blows up if you just do that. Many packages work with Python2 or Python3; however, when built with Python2, they require Python2-only support libraries that are backports of standard Python3 features. The Spack packages aren't smart enough to eliminate these dependencies when building with Python3, thereby making the Spack recipe (but not the upstream Python library) not work with Python3. This issue must be ironed out before we can talk seriously about changing Python version defaults. Either we run with something like what we have in #7926; or we fix the concretizer so we can do stuff like depends_on('py-backport', when='^python@2').

Summary: before we can switch the default version to Python3, we need to make sure that Python3-based stacks build smoothly without modifying / patching / merging in from other branches.

As for the Python2-only packages identified above, someone needs to do the research and categorize them as:

  1. A backport that is not needed in Python3.
  2. A Python2-only package that has been made obsolete by some other Python3-only package (and identify what that Python3 package is).
  3. A package that was Python2-only at some point; but now also works with Python3.
  4. A true Python2-only package, with no functional replacement for Python3.
  5. Non-Python packages that happen to use Python2 scripts to build or something; and nobody has bothered to convert them to Python3.

At this point, there aren't many packages in category (4). My guess is they are mostly Python interfaces to specialized domain-specific software with only a small number of users. For example, PISM only supported Python2 bindings until Dec 2018. Apparently, PETSc still requires Python2 to build (although it's not a Python package per se, so it falls in category 5).
pism/pism#359

ISSM is another example. They mostly use MATLAB (not Python) in-house anyway. ISSM is also still running on SVN (not git). And they hand-build 50+ packages as prerequisites, rather than using Spack. So I wouldn't expect Python3 bindings any time soon. OTOH, ISSM is specialty software with a small user base.
https://issm.jpl.nasa.gov/download/unix/

@adamjstewart
Copy link
Member Author

The tut build tests are failing. After looking into this, it looks like the version of waf shipped with tut does not support Python 3.7. This bug was fixed upstream in waf but needs to be updated in tut:
https://gitlab.com/ita1024/waf/commit/facdc0b173d933073832c768ec1917c553cb369c

@citibeth
Copy link
Member

citibeth commented Jan 15, 2019 via email

@adamjstewart adamjstewart mentioned this pull request Feb 1, 2019
Closed
@citibeth
Copy link
Member

citibeth commented Mar 6, 2019

Note the list of packages that declare they will require Python3, starting in the next year or so:
https://python3statement.org/

This includes all the basics: numpy, matplotlib, basemap, scipy, ipython, jupyter, pandas. I think that sometime in 2019 is the time for Spack to commit to Python3 by default.

@citibeth citibeth mentioned this pull request May 15, 2019
Closed
This was referenced May 23, 2019
Closed
Closed
@baberlevi
Copy link
Member

baberlevi commented May 23, 2019 •
edited
Loading

I'm trying to understand what's holding back this PR (besides the tut build test). The way I'm looking at it, the packages that support python2-only aren't negatively affected by this change - they already have a dependency on python2 explicitly and concretize correctly even with the PR in place. The python3-only packages are likewise fine with this PR. The packages that are some combination of depends_on(python@2:2.9,3:) are the ones currently 'broken' in the out-of-the-box user experience, and would be 'fixed' again with this PR.

@citibeth
Copy link
Member

Lack of a coherent agreed-upon plan is holding us back for now. See #11468.

@scheibelp scheibelp mentioned this pull request Jun 18, 2019
Closed
@adamjstewart
Copy link
Member Author

The tut build tests are failing. After looking into this, it looks like the version of waf shipped with tut does not support Python 3.7. This bug was fixed upstream in waf but needs to be updated in tut:
https://gitlab.com/ita1024/waf/commit/facdc0b173d933073832c768ec1917c553cb369c

Filed an issue: mrzechonek/tut-framework#18

@adamjstewart
Copy link
Member Author

@citibeth does pism support Python 3 yet?

@adamjstewart
Copy link
Member Author

adamjstewart commented Jul 22, 2019 •
edited
Loading

Went through all the packages in Spack that claim to only support Python 2. A few now have Python 3 support, but the vast majority are either a work in progress or are no longer supported. A few like julia are important enough that we might want to wait for Python 3 support.

@citibeth
Copy link
Member

citibeth commented Jul 22, 2019 via email

@adamjstewart adamjstewart mentioned this pull request Jul 26, 2019
Merged
@adamjstewart
Copy link
Member Author

#12145 may put the brakes on this PR for awhile. The latest version of Python 3 can't be built with the Intel compilers. Hopefully they will come out with a patch soon.

@adamjstewart
Copy link
Member Author

adamjstewart commented Jul 28, 2019 •
edited
Loading

Note that we've reached the point where the latest releases of numpy, scipy, matplotlib, pandas, and scikit-learn no longer support Python 2. This will continue to grow rapidly: https://python3statement.org/

@svenevs
Copy link
Member

svenevs commented Jul 28, 2019 

- no longer support Python 3
+ no longer support Python 2

😉

@sethrj
Copy link
Contributor

sethrj commented Jul 28, 2019

Also note that py-pil has been essentially superseded by Pillow, which does have Python 3 support and in fact will soon by dropping support for 2.7.

@goxberry
Copy link
Contributor

@adamjstewart Elemental is unmaintained. There have been a couple maintenance patches here and there, but the author has explicitly stated he's not working on it anymore, and to the best of my knowledge, no one has stepped up to maintain it.

As far as I know, Hydrogen (as in, the Spack package) forks Elemental, but last I checked, Hydrogen strips out everything except BLAS level-1 primitives, and then adds GPU support. (I could be wrong about the specifics, but conversations with Hydrogen maintainers left me with this overall impression.) I think dropping support for Elemental during a migration to Python 3 would be an entirely reasonable position by the Spack maintainers, but I don't know how many people would be affected by such a change, and how much it would affect them.

@adamjstewart
Copy link
Member Author

I think dropping support for Elemental during a migration to Python 3 would be an entirely reasonable position by the Spack maintainers, but I don't know how many people would be affected by such a change, and how much it would affect them.

Just to clarify, a switch to Python 3 by default won't drop support for anything. Users can still specify spack install elemental ^python@:2 until the concretizer is fixed. Once the concretizer is fixed, users won't even notice that the new default is Python 3.

@adamjstewart
Copy link
Member Author

Can we hold off until we can ensure that a decent subset of the python 3 packages build fine as binaries?

@tgamblin What do you consider to be a decent subset? Most core Python libraries (numpy, scipy, matplotlib, pandas, sklearn) only support Python 3 at this point. I use all of these in my research and haven't run into any problems so far. I'm willing to test a larger subset if that's what it takes to get this PR merged.

@adamjstewart adamjstewart mentioned this pull request Oct 11, 2019
Closed
@adamjstewart adamjstewart mentioned this pull request Oct 20, 2019
Merged
@adamjstewart adamjstewart changed the title Default to Python3 Default to Python 3.7 Oct 20, 2019
@adamjstewart
Copy link
Member Author

Python 3.8.0 was recently released. I've been playing around with it, and it's horribly buggy so far (see #13256). Based on this, and the fact that Anaconda and Homebrew still don't offer Python 3.8 support, I've decided to switch the default to Python 3.7 instead of Python 3. I propose we move in lockstep with Anaconda, as they are kind of the authority on Python distributions these days.

@citibeth
Copy link
Member

citibeth commented Oct 20, 2019 via email

@adamjstewart
Copy link
Member Author

We're finally seeing progress on fixing the Python build for the Intel compilers! python/cpython#16717

If this patch works, we can remove the conflict and this PR should be ready to merge.

@adamjstewart
Copy link
Member Author

Can we hold off until we can ensure that a decent subset of the python 3 packages build fine as binaries?

@tgamblin can you define a decent subset? I've been using the following packages with Python 3.7.4 without any problems:

$ spack find
==> 200 installed packages
-- darwin-catalina-ivybridge / clang@11.0.0-apple ---------------
apr@1.7.0              jackcess@1.2.14.3    openssl@1.1.1d                 py-hypothesis@4.41.2        py-pyparsing@2.4.2                      py-torch@1.3.0
apr-util@1.6.1         jasper@1.900.1       perl@5.30.0                    py-idna@2.8                 py-pysocks@1.7.1                        py-torchvision@0.4.0
autoconf@2.69          jasper@2.0.14        pkgconf@1.6.3                  py-imagesize@1.1.0          py-pytest@5.2.1                         py-tornado@6.0.3
automake@1.16.1        jdk@12.0.1_12        proj@6.2.0                     py-importlib-metadata@0.23  py-pytest-cov@2.8.1                     py-typed-ast@1.4.0
bash@5.0               json-c@0.13.1        py-alabaster@0.7.12            py-itsdangerous@1.1.0       py-pytest-forked@1.1.1                  py-typing@3.7.4.1
binutils@2.32          libffi@3.2.1         py-apipkg@1.5                  py-jinja2@2.10.3            py-pytest-httpbin@0.0.7                 py-typing-extensions@3.7.2
bison@3.4.2            libgcrypt@1.8.4      py-atomicwrites@1.3.0          py-joblib@0.14.0            py-pytest-httpbin@1.0.0                 py-typing-extensions@3.7.4
boost@1.71.0           libgcrypt@1.8.5      py-attrs@19.2.0                py-kiwisolver@1.1.0         py-pytest-mock@1.11.1                   py-urllib3@1.25.6
bzip2@1.0.8            libgeotiff@1.5.1     py-babel@2.7.0                 py-lxml@4.4.1               py-pytest-runner@5.1                    py-virtualenv@16.7.6
c-blosc@1.17.0         libgpg-error@1.36    py-blinker@1.4                 py-markupsafe@1.1.1         py-pytest-xdist@1.30.0                  py-wcwidth@0.1.7
c-blosc@1.17.0         libiconv@1.16        py-bottleneck@1.2.1            py-matplotlib@3.1.1         py-python-dateutil@2.8.0                py-webencodings@0.5.1
cmake@3.15.4           libjpeg-turbo@2.0.3  py-brotlipy@0.7.0              py-mccabe@0.6.1             py-pytz@2019.3                          py-werkzeug@0.16.0
commons-lang@2.4       libkml@1.3.0         py-certifi@2019.9.11           py-mock@3.0.5               py-pyyaml@5.1.2                         py-zipp@0.6.0
commons-logging@1.1.1  libpng@1.6.37        py-cffi@1.13.0                 py-more-itertools@7.2.0     py-raven@6.10.0                         python@3.7.4
diffutils@3.7          libsigsegv@2.12      py-chardet@3.0.4               py-mypy@0.740               py-requests@2.22.0                      readline@8.0
expat@2.2.9            libtiff@4.0.10       py-click@6.6                   py-mypy-extensions@0.4.3    py-scikit-learn@0.21.3                  serf@1.3.9
flex@2.6.4             libtool@2.4.6        py-click@7.0                   py-numexpr@2.7.0            py-scipy@1.3.1                          snappy@1.1.7
freetype@2.10.1        libxml2@2.9.9        py-coverage@4.5.4              py-numexpr@2.7.0            py-setuptools@41.4.0                    sqlite@3.30.0
gcc@9.2.0              libxslt@1.1.33       py-cycler@0.10.0               py-numpy@1.17.3             py-setuptools-scm@3.3.3                 subversion@1.12.2
gdal@3.0.1             libyaml@0.2.2        py-cython@0.29.13              py-numpy@1.17.3             py-six@1.12.0                           tar@1.32
gdbm@1.18.1            llvm-openmp@9.0.0    py-decorator@4.4.0             py-packaging@19.2           py-snowballstemmer@2.0.0                uriparser@0.9.3
gettext@0.20.1         lz4@1.9.2            py-docutils@0.14               py-pandas@0.25.1            py-sphinx@2.2.0                         utf8proc@2.4.0
gmake@4.2.1            m4@1.4.18            py-docutils@0.15.2             py-pillow@6.2.0             py-sphinx-rtd-theme@0.4.3               watch@3.3.15
gmp@6.1.2              mercurial@5.1.2      py-docutils-stubs@0.0.21       py-pip@19.3                 py-sphinxcontrib-applehelp@1.0.1        wget@1.20.3
googletest@1.8.1       minizip@1.2.11       py-entrypoints@0.3             py-pluggy@0.13.0            py-sphinxcontrib-devhelp@1.0.1          xz@5.2.4
googletest@1.10.0      mpc@1.1.0            py-execnet@1.7.1               py-psutil@5.6.3             py-sphinxcontrib-htmlhelp@1.0.2         zlib@1.2.11
graphviz@2.42.2        mpfr@3.1.6           py-flake8@3.7.8                py-py@1.8.0                 py-sphinxcontrib-jsmath@1.0.1           zstd@1.4.3
hdf@4.2.14             nasm@2.14.02         py-flake8-import-order@0.18.1  py-pycodestyle@2.5.0        py-sphinxcontrib-qthelp@1.0.2           zstd@1.4.3
hdf5@1.10.5            ncurses@6.1          py-flask@1.1.1                 py-pycparser@2.18           py-sphinxcontrib-serializinghtml@1.1.3
help2man@1.47.11       ninja@1.9.0          py-freezegun@0.3.12            py-pycparser@2.19           py-sphinxcontrib-websupport@1.1.2
intel-mkl@2019.4       openblas@0.3.7       py-html5lib@1.0.1              py-pyflakes@2.1.1           py-tables@3.6.0
isl@0.20               openjpeg@2.3.1       py-httpbin@0.7.0               py-pygments@2.4.2           py-tables@3.6.0

In addition, the following packages ONLY support Python 3: https://python3statement.org/

@tgamblin
Copy link
Member

@adamjstewart: that looks like a decent subset to me. I agree with picking 3.7 instead of 3.8.

The only other question, which I had for @scheibelp and I believe he fixed in #13243, is whether things will still continue to work as well as can be expected if you prefer python 2.7 in packages.yaml. I'm pretty sure that will continue working fine.

If you agree, I'm totally fine with this and think it's the right way to go.

@adamjstewart
Copy link
Member Author

@tgamblin I've been preferring 3.7 in my packages.yaml for a couple years now, so I imagine preferring 2.7 should work the same. The only problem is that the concretizer still isn't capable of determining the latest version of numpy/scipy/matplotlib/etc that supports Python 2.7.

Let me test python/cpython#16717 first. If that patch works, I'll add it to this PR and we can finally merge this!

@tgamblin
Copy link
Member

The only problem is that the concretizer still isn't capable of determining the latest version of numpy/scipy/matplotlib/etc that supports Python 2.7.

Exactly - I believe this is what #13243 was meant to fix. @scheibelp? I am hoping to have an "experimental" new concretizer in sometime this month so that people can start trying it when they have these issues.

@scheibelp
Copy link
Member

scheibelp commented Oct 21, 2019 •
edited
Loading

The only problem is that the concretizer still isn't capable of determining the latest version of numpy/scipy/matplotlib/etc that supports Python 2.7.

Exactly - I believe this is what #13243 was meant to fix. scheibelp?

I don't think so: If you want to support something like

spack spec py-numpy ^python@:2

and you want Spack to choose the latest version of py-numpy that can build against Python 2.x, then #13243 will not help the concretizer find the appropriate constraints. The result would most likely be a failure, since those packages will need to encode constraints like

depends_on(python@3:, when=@x:)

I didn't consider this a major problem since a user could explicitly specify the version of py-numpy that they want to build along with Python 2.x like

spack spec py-numpy@x ^python@:2

As for what #13243 fixes: if a package only needs some dependencies for Python 2.x versions, #13243 fixes a bug where such constraints were not respected. I think that was a more serious bug because it led Spack to add dependencies that weren't needed (i.e. the concretization completed without errors but produced an incorrect DAG).

So my question is: do we consider it a blocker that users who want py-numpy with python 2.x will have to specify a particular version? I think if so, we could recommend a configuration that sets preferences on some py-* packages that allows them to build with Python 2.x

@citibeth
Copy link
Member

citibeth commented Oct 21, 2019 via email

@scheibelp
Copy link
Member

So my question is: do we consider it a blocker that users who want py-numpy with python 2.x will have to specify a particular version? I think if so, we could recommend a configuration that sets preferences on some py-* packages that allows them to build with Python 2.x

Maybe it's time to use the concretizer workaround I proposed earlier with the +python2 variant?

I think you're referring to #11468 and #7926? Some of the changes I see in #7926 e.g.

PyMatplotlib():
...
    depends_on('py-subprocess32', type=('build', 'run'), when='~python3')
    depends_on('py-functools32', type=('build', 'run'), when='~python3')

are resolved by #13243 e.g.

PyMatplotlib():
...
    depends_on('py-subprocess32', type=('build', 'run'), when='^python@:2')
    depends_on('py-functools32', type=('build', 'run'), when='^python@:2')

To support spack spec py-numpy ^python@:2 more-smoothly (and again I think it's a matter of debate whether it is considered a blocker at all) I think a specialized config file could be added to Spack's etc/ directory to select all the appropriate preferred python versions for Python 2.x (and the user could activate it with spack -C etc/py2-config/ ...).

@adamjstewart
Copy link
Member Author

So my question is: do we consider it a blocker that users who want py-numpy with python 2.x will have to specify a particular version? I think if so, we could recommend a configuration that sets preferences on some py-* packages that allows them to build with Python 2.x

So you're asking if the fact that this fails:

$ spack install py-numpy ^python@:2

should be a blocker to merging this PR? What about the fact that without this PR, the following fails:

$ spack install py-numpy

I consider that a much more serious issue.

@scheibelp
Copy link
Member

scheibelp commented Oct 21, 2019 •
edited
Loading

What about the fact that without this PR, the following fails:
$ spack install py-numpy
I consider that a much more serious issue.

I agree with you.

You made a statement earlier:

The only problem is that the concretizer still isn't capable of determining the latest version of numpy/scipy/matplotlib/etc that supports Python 2.7.

I assume you were implying that the following would fail to concretize with this PR:

spack spec py-numpy ^python@:2

which I think is true but I also don't think it's a blocker. I was asking if other people thought it was. In particular I wanted to ask because in #10319 (comment) it was asked whether #13243 would resolve this problem; it does not, so the issue you brought up would remain an issue with the PR (but again I don't think it's a blocker).

@adamjstewart
Copy link
Member Author

Gotcha. I personally don't consider it a blocker, since you already have to specify the correct version of numpy that supports Python 2 with or without this PR. I think we can wait on the new concretizer to resolve that issue.

@tgamblin
Copy link
Member

I agree completely with:

What about the fact that without this PR, the following fails:

$ spack install py-numpy

I consider that a much more serious issue.

and:

I personally don't consider it a blocker, since you already have to specify the correct version of numpy that supports Python 2 with or without this PR.

@adamjstewart: Let's go ahead and merge this.

@adamjstewart
Copy link
Member Author

The patch worked! I was able to successfully installs Python 3.7.4 with Intel 18.0.3.222 on Cray CNL5. I think this PR is finally ready to merge as soon as Travis passes.

@adamjstewart adamjstewart merged commit 526ee11 into spack:develop Oct 22, 2019
@adamjstewart adamjstewart deleted the packages/python3 branch October 22, 2019 15:22
@adamjstewart
Copy link
Member Author

Thanks for everyone's help in getting this PR merged!

If you still rely on packages that require Python 2, please encourage the developers to update to Python 3. If the package should support Python 3 but has build problems, let me know and I'll take a look. If you want to restore the behavior before this PR, make the following change to your packages.yaml:

packages:
  python:
    version: [':2']

R.I.P. Python 2, long live Python 3!

@healther healther mentioned this pull request Oct 24, 2019
Closed
jrmadsen pushed a commit to jrmadsen/spack that referenced this pull request Oct 30, 2019
@adamjstewart @jrmadsen
Default to Python 3.7 (spack#10319)
6ec52b4 
* Default to Python 3

* Fix build with Intel compilers
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@scheibelp scheibelp

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Python3 Migration Strategy
10 participants
@adamjstewart @tgamblin @citibeth @baberlevi @svenevs @sethrj @goxberry @jabl @healther @scheibelp