Installation via pip from PyPi with Python=3.9 doesn't work

[valeriupredoi]

Hey guys, apologies for spamming, but I am following the advice given to me by @rbuffat to open an issue that I initially had only a couple comments for in #978 see comment

Summary
pip install fiona does not give a working installation when using python=3.9 in a virtual conda environment; this used to work fine before for lower versions of Python; workaround for us, since we must use a conda env for our package's dependencies (see the reasons in my comment above) is to install fiona from conda-forge

Details

(esmvalcore39) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ pip install fiona
Collecting fiona
  Using cached Fiona-1.8.18.tar.gz (1.3 MB)
Requirement already satisfied: attrs>=17 in /home/valeriu/miniconda3/envs/esmvalcore39/lib/python3.9/site-packages (from fiona) (20.3.0)
Requirement already satisfied: certifi in /home/valeriu/miniconda3/envs/esmvalcore39/lib/python3.9/site-packages (from fiona) (2020.11.8)
Requirement already satisfied: click<8,>=4.0 in /home/valeriu/miniconda3/envs/esmvalcore39/lib/python3.9/site-packages (from fiona) (7.1.2)
Requirement already satisfied: cligj>=0.5 in /home/valeriu/miniconda3/envs/esmvalcore39/lib/python3.9/site-packages (from fiona) (0.7.1)
Requirement already satisfied: click-plugins>=1.0 in /home/valeriu/miniconda3/envs/esmvalcore39/lib/python3.9/site-packages (from fiona) (1.1.1)
Requirement already satisfied: six>=1.7 in /home/valeriu/miniconda3/envs/esmvalcore39/lib/python3.9/site-packages (from fiona) (1.15.0)
Requirement already satisfied: munch in /home/valeriu/miniconda3/envs/esmvalcore39/lib/python3.9/site-packages (from fiona) (2.5.0)
Building wheels for collected packages: fiona
  Building wheel for fiona (setup.py) ... done
  Created wheel for fiona: filename=Fiona-1.8.18-cp39-cp39-linux_x86_64.whl size=767781 sha256=d885bdb178d6dadab316b056c558bfac52f232a3afca31b11003bc3a714cbb77
  Stored in directory: /home/valeriu/.cache/pip/wheels/00/af/d5/91e40f5a48370c4dad3552d69bebc0b786f6715793db861d42
Successfully built fiona
Installing collected packages: fiona
Successfully installed fiona-1.8.18
(esmvalcore39) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ fio --version
Traceback (most recent call last):
  File "/home/valeriu/miniconda3/envs/esmvalcore39/bin/fio", line 5, in <module>
    from fiona.fio.main import main_group
  File "/home/valeriu/miniconda3/envs/esmvalcore39/lib/python3.9/site-packages/fiona/__init__.py", line 86, in <module>
    from fiona.collection import BytesCollection, Collection
  File "/home/valeriu/miniconda3/envs/esmvalcore39/lib/python3.9/site-packages/fiona/collection.py", line 11, in <module>
    from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
ImportError: libgdal.so.26: cannot open shared object file: No such file or directory
(esmvalcore39) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ python -V
Python 3.9.0
(esmvalcore39) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ pip --version
pip 20.2.4

Possible solution

It seems that gdal and libgdal are not installed and built as dependencies when the pip installation is called - your CI GA tests pass fine, I can see them, but that's because you building GDAL before you are building fiona - is this now a requirement for the pip installation? We used the pip install fine with Python=3.8 in our conda environment before, without the need to build GDAL and fiona worked fine:

(esmvalcore38) [valeriu@sci2 esmvalcore]$ python -V
Python 3.8.6
(esmvalcore38) [valeriu@sci2 esmvalcore]$ conda list fiona
# packages in environment at /home/users/valeriu/anaconda3R/envs/esmvalcore38:
#
# Name                    Version                   Build  Channel
fiona                     1.8.18                   pypi_0    pypi
(esmvalcore38) [valeriu@sci2 esmvalcore]$ conda list gdal
# packages in environment at /home/users/valeriu/anaconda3R/envs/esmvalcore38:
#
# Name                    Version                   Build  Channel
(esmvalcore38) [valeriu@sci2 esmvalcore]$ fio --version
fio, version 1.8.18

Cheers muchly for your attention 🍺

[sgillies]

First, we're not support pip install into conda environments. It has only worked before by accident. Second, the actual problem is lack of a 3.9 binary wheel on PyPI, whereas there is a binary wheel for 3.8.

That said, I did find a little time last night to add a manylinux1 py39 job to the fiona-wheels build matrix and I published a wheel to PyPI today. Success with osx evades me, we'll eventually have an osx wheel too.

[valeriupredoi]

@sgillies cheers muchly!

That said, I did find a little time last night to add a manylinux1 py39 job to the fiona-wheels build matrix and I published a wheel to PyPI today. Success with osx evades me, we'll eventually have an osx wheel too.

I can confirm that the installation now works well in a conda environment too with fiona=1.8.18 from PyPi, good work on the wheels! If I may suggest, this way of installing fiona (in a conda env from PyPi) is a good thing to have - offers more flexibility to groups like ours where we need to use conda envs (sadly) but we'd like as many deps as possible to be installed from PyPi for a more robust environment solvability and dependency management; so if you guys do the same for the next releases of fiona it'd be awesome! Cheers again for your help 🍺