Working environment for Python 3.9 (pin to !=3.9.0)

[valeriupredoi]

Changes applied (so far):

• relaxed (removed) pin on python<3.9
• moved fiona to be installed from conda-forge due to this comment - also see the issue I opened in Fiona Installation via pip from PyPi with Python=3.9 doesn't work Toblerity/Fiona#983
  Issues:
• need to figure out why the esmpy regridder tests are failing (3 tests)

Closes #881

[valeriupredoi]

OK a few notes:

• I reverted back to fiona installed from PyPi (they have built wheels for manylinux now, but I am not sure how reliable this will be in the future, we may have to switch to installing it from conda-forge since they put it bluntly that they don't support PyPi installations inside conda environments)
• the mock error is weird, I documented and raised it here mock throws ValueError when spec_arg is an ndarray testing-cabal/mock#493

[valeriupredoi]

OK I have now isolated the issue with unittest.mock - it doesn't like an ndarray spec arg - pretty straightforward, it likes every other type tho:

import numpy as np

from unittest import mock as mg

ob = mg.Mock(spec=[4, 4])
print("spec is list, mock is", ob)

ob = mg.Mock(spec=["4", "4"])
print("spec is list of str, mock is", ob)

ob = mg.Mock(spec=(4, 4))
print("spec is tuple, mock is", ob)

ob = mg.Mock(spec="cow")
print("spec is string, mock is", ob)

ob = mg.Mock(spec=22)
print("spec is int, mock is", ob)

ob = mg.Mock(spec=np.array([4, 4]))
print("spec is ndarray, mock is", ob)

I am going to open a bug report on Her Majesty Python

[valeriupredoi]

OK issue open in Python: https://bugs.python.org/issue42586

[valeriupredoi]

OK I have now isolated the issue with unittest.mock - it doesn't like an ndarray spec arg - pretty straightforward, it likes every other type tho:

import numpy as np

from unittest import mock as mg

ob = mg.Mock(spec=[4, 4])
print("spec is list, mock is", ob)

ob = mg.Mock(spec=["4", "4"])
print("spec is list of str, mock is", ob)

ob = mg.Mock(spec=(4, 4))
print("spec is tuple, mock is", ob)

ob = mg.Mock(spec="cow")
print("spec is string, mock is", ob)

ob = mg.Mock(spec=22)
print("spec is int, mock is", ob)

ob = mg.Mock(spec=np.array([4, 4]))
print("spec is ndarray, mock is", ob)

I am going to open a bug report on Her Majesty Python

@bouweandela @zklaus you guys have more experience with mock than meself, is this something normal (also see our esmpy regrid tests that fail coz of this)? 🍺

[valeriupredoi]

OK proves out that was inded a bug in 3.9.0 that was/is being fixed and will be in 3.9.1, so we'll just hold off using 3.9.0 and wait for 3.9.1 that already has an RC and looks like it'll roll out on Friday, 11 Dec - see full discussion https://bugs.python.org/issue42586 and the Python schedules at https://www.python.org/dev/peps/pep-0596/#lifespan

[zklaus]

Thanks for checking this, @valeriupredoi!

[valeriupredoi]

big day for Python 3.9.1 release tomorrow @zklaus 😁 Hopefully by next week we'll have a very modern and very working environment hehe

[valeriupredoi]

an update to this one: the environment now builds and does it with python 3.9.1 unfortunately the new version from conda doesn't include the patch for mock.py:

python                    3.9.1           hffdb5ce_2_cpython    conda-forge

is the latest off conda-forge (uploaded on December 21st, conda list) and should contain the updated library mock with the fix at l.409 - but no, I don't know why it didn't make it - possibly next version, agh

[bouweandela]

Maybe you could just mark the failing tests as should be skipped in python 3.9?

[valeriupredoi]

no coz the fix is there, proves out that they've not released 3.9.x after December 7th https://github.com/python/cpython/releases
so my bets are 3.9.2 will work

[bouweandela]

So maybe you can just mark the test for skipping with python 3.9.0 and 3.9.1? If the only problem is some unit test failing because of the test code (not because of a problem with code in the ESMValCore), we can probably just start supporting Python 3.9.

[valeriupredoi]

Maybe you could just mark the failing tests as should be skipped in python 3.9?

OK - maybe the best solution since they defo don't work in 3.9.0 and 3.9.1 and even if they will work in 3.9.2 it's still 3.9.x - how do you think - xfail them or skip them?

[valeriupredoi]

So maybe you can just mark the test for skipping with python 3.9.0 and 3.9.1? If the only problem is some unit test failing because of the test code (not because of a problem with code in the ESMValCore), we can probably just start supporting Python 3.9.

yeah was just writing my comment at the same time 😁

[valeriupredoi]

OK things are in good shape, apart from a weird test fail (that is for Python 3.8 too, documented in #927 ) and lots of segfaults and the obvious issue that you can't install esmvalcore from pip with Python=3.9 (since it's a Python 3.8 build on pip), the GA tests and the ones on Circle are OK https://github.com/ESMValGroup/ESMValCore/actions (look for "test GA for Python 3.9" eg these) - I think we can do 2.2.0 with Python 3.9 easily -> @jvegasbsc

[valeriupredoi]

right - this should be ready for merge! All major (non-silly, like installing esmvalcore from conda or pypi in a Python 3.9 environment, with the builds being done with Python 3.8) tests and builds work! 🍺

[valeriupredoi]

@schlunma @jvegasbsc @stefsmeets @bouweandela I need an approval of this the sooner the better guys please - we should have support for Python 3.9 for the 2.2 release and it'd be better to have a few days in mainstream before we make the release just to be sure all stuff's working well 👍 🍺

[stefsmeets]

Looks good, no obvious red flags!👍
(Soon enough? 😅)

[valeriupredoi]

cheers @stefsmeets mate 🍺