Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

conda-lock pip dependency gets .tar.gz sdist rather than manylinux wheel when available manylinux wheel is more recent than 2_17 #517

Closed
2 tasks done
lesteve opened this issue Sep 29, 2023 · 0 comments · Fixed by #541
Closed
2 tasks done

conda-lock pip dependency gets .tar.gz sdist rather than manylinux wheel when available manylinux wheel is more recent than 2_17 #517

lesteve opened this issue Sep 29, 2023 · 0 comments · Fixed by #541

Comments

@lesteve
Copy link
Contributor

lesteve commented Sep 29, 2023 •
edited
Loading

Checklist

  • I added a descriptive title
  • I searched open reports and couldn't find a duplicate

What happened?

.tar.gz sdist are found instead of .whl when manylinux wheel are more recent than 2_17. This can cause long compilation on the machine doing conda-lock install. I believe #395 has exactly the same root cause since the .whl in this case is a 2_27 and is not found by conda-lock and there is no sdist available on PyPI for open3d.

To reproduce:

# environment.yml
channels:
  - conda-forge

dependencies:
  - pip
  - pip:
    - lightgbm

conda-lock lock -f environment.yml --platform linux-64

The lockfile has a sdist .tar.gz rather than a .whl.

❯ grep lightgbm conda-lock.yml
- name: lightgbm
  url: https://files.pythonhosted.org/packages/98/a9/01f50aee85949ba713733b69a3f0b42d39719a414a0e29bdf2a9f05ecc53/lightgbm-4.1.0.tar.gz

lightgm has some manylinux_2_28 wheels that I can install fine on my Linux machine with pip: https://pypi.org/project/lightgbm/4.1.0/#files

I think this comes from the fact that the possible manylinux wheels recognised by conda-lock are hard-coded here (up until 2_17):

conda-lock/conda_lock/pypi_solver.py

Line 42 in f64b74e

MANYLINUX_TAGS = ["1", "2010", "2014", "_2_17"]

It is a bit tricky to know which manylinux wheels should be supported, since the ones supported by the machine running conda-lock lock are not necessarily supported by the machine running conda-lock install. Maybe some heuristics could be used like all the manylinux wheels until the most recent docker image supported by manylinux? Right now it is 2_28 according to https://github.com/pypa/manylinux/blob/main/README.rst#docker-images.

Conda Info

active environment : conda-lock
    active env location : /home/lesteve/micromamba/envs/conda-lock
            shell level : 1
       user config file : /home/lesteve/.condarc
 populated config files : /home/lesteve/dev/dotfiles/.condarc
          conda version : 23.7.2
    conda-build version : not installed
         python version : 3.11.4.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.38=0
                          __linux=6.5.3=0
                          __unix=0=0
       base environment : /home/lesteve/micromamba  (writable)
      conda av data dir : /home/lesteve/micromamba/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/lesteve/micromamba/pkgs
                          /home/lesteve/.conda/pkgs
       envs directories : /home/lesteve/micromamba/envs
                          /home/lesteve/.conda/envs
               platform : linux-64
             user-agent : conda/23.7.2 requests/2.31.0 CPython/3.11.4 Linux/6.5.3-arch1-1 endeavouros/rolling glibc/2.38
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

Conda Config

==> /home/lesteve/dev/dotfiles/.condarc <==
changeps1: False
channels:
  - conda-forge

==> cmd_line <==
debug: False
json: False

Conda list

# packages in environment at /home/lesteve/micromamba/envs/conda-lock:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
brotlipy                  0.7.0           py311hd4cff14_1005    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cachecontrol              0.13.0             pyhd8ed1ab_0    conda-forge
cachecontrol-with-filecache 0.13.0             pyhd8ed1ab_0    conda-forge
cachy                     0.3.0              pyhd8ed1ab_1    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py311h409f033_3    conda-forge
charset-normalizer        3.1.0              pyhd8ed1ab_0    conda-forge
click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
click-default-group       1.2.2              pyhd8ed1ab_1    conda-forge
clikit                    0.6.2              pyhd8ed1ab_2    conda-forge
conda-lock                2.3.0              pyhd8ed1ab_0    conda-forge
crashtest                 0.4.1              pyhd8ed1ab_0    conda-forge
cryptography              41.0.1          py311h63ff55d_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
distlib                   0.3.6              pyhd8ed1ab_0    conda-forge
ensureconda               1.4.3              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
filelock                  3.12.2             pyhd8ed1ab_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
gitdb                     4.0.10             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.31             pyhd8ed1ab_0    conda-forge
html5lib                  1.1                pyh9f0ad1d_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.7.0              pyha770c72_0    conda-forge
importlib_metadata        6.7.0                hd8ed1ab_0    conda-forge
jaraco.classes            3.2.3              pyhd8ed1ab_0    conda-forge
jeepney                   0.8.0              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
keyring                   23.13.1         py311h38be061_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libglib                   2.76.3               hebfc3b9_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lockfile                  0.12.2                     py_1    conda-forge
markupsafe                2.1.3           py311h459d7ec_0    conda-forge
more-itertools            9.1.0              pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.5           py311ha3edf6b_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
openssl                   3.1.3                hd590300_0    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pastel                    0.2.1              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pip                       23.1.2             pyhd8ed1ab_0    conda-forge
pkginfo                   1.9.6              pyhd8ed1ab_0    conda-forge
platformdirs              3.6.0              pyhd8ed1ab_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  1.10.9          py311h459d7ec_0    conda-forge
pylev                     1.4.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.4          hab00c5b_0_cpython    conda-forge
python_abi                3.11                    3_cp311    conda-forge
pyyaml                    6.0             py311hd4cff14_5    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.17.32         py311h459d7ec_0    conda-forge
ruamel.yaml.clib          0.2.7           py311h2582759_1    conda-forge
secretstorage             3.3.3           py311h38be061_1    conda-forge
setuptools                67.7.2             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     3.0.5              pyh44b312d_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tomlkit                   0.11.8             pyha770c72_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
typing-extensions         4.6.3                hd8ed1ab_0    conda-forge
typing_extensions         4.6.3              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
virtualenv                20.23.1            pyhd8ed1ab_0    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zipp                      3.15.0             pyhd8ed1ab_0    conda-forge

Additional Context

No response
@lesteve lesteve changed the title conda-lock pip dependency gets sdist .tar.gz rather than manylinux wheel when available manylinux wheel is more recent than 2_17 conda-lock pip dependency gets .tar.gz sdist rather than manylinux wheel when available manylinux wheel is more recent than 2_17 Sep 29, 2023
@lesteve lesteve mentioned this issue Oct 27, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make recent manylinux wheels findable by pypi_resolver lesteve/conda-lock
1 participant
@lesteve