Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error creating lockfile from rendered environment #624

Closed
2 tasks done
sfinkens opened this issue Apr 19, 2024 · 6 comments · Fixed by #629
Closed
2 tasks done

Error creating lockfile from rendered environment #624

sfinkens opened this issue Apr 19, 2024 · 6 comments · Fixed by #629

Comments

@sfinkens
Copy link
Contributor

sfinkens commented Apr 19, 2024 •
edited
Loading

Checklist

  • I added a descriptive title
  • I searched open reports and couldn't find a duplicate

What happened?

conda-lock fails to create a lockfile from a rendered environment that includes pip dependencies. To reproduce:

  1. Create environment.yml 
    channels:
  - conda-forge
dependencies:
  - python
  - pip:
    - flit-core
  2. Create lockfile: conda-lock -p linux-64 -f environment.yml
  3. Render environment: conda-lock render -p linux-64 -k env conda-lock.yml
  4. Create lockfile from rendered environment: conda-lock -p linux-64 -f conda-linux-64.lock.yml

This fails with the following traceback.

Traceback 
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "###/bin/conda-lock", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "###/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "###/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "###/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "###/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "###/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "###/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "###/lib/python3.12/site-packages/conda_lock/conda_lock.py", line 1402, in lock
    lock_func(
  File "###/lib/python3.12/site-packages/conda_lock/conda_lock.py", line 1110, in run_lock
    make_lock_files(
  File "###/lib/python3.12/site-packages/conda_lock/conda_lock.py", line 337, in make_lock_files
    lock_spec = make_lock_spec(
                ^^^^^^^^^^^^^^^
  File "###/python3.12/site-packages/conda_lock/src_parser/__init__.py", line 91, in make_lock_spec
    lock_specs = _parse_source_files(src_files, platforms)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "###/lib/python3.12/site-packages/conda_lock/src_parser/__init__.py", line 71, in _parse_source_files
    desired_envs.append(parse_environment_file(src_file, platforms))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "###/lib/python3.12/site-packages/conda_lock/src_parser/environment_yaml.py", line 133, in parse_environment_file
    platform: _parse_environment_file_for_platform(content, category, platform)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "###/lib/python3.12/site-packages/conda_lock/src_parser/environment_yaml.py", line 73, in _parse_environment_file_for_platform
    parse_python_requirement(
  File "###/lib/python3.12/site-packages/conda_lock/src_parser/pyproject_toml.py", line 423, in parse_python_requirement
    parsed_req = parse_requirement_specifier(requirement)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "###/lib/python3.12/site-packages/conda_lock/src_parser/pyproject_toml.py", line 397, in parse_requirement_specifier
    return Requirement(requirement)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "###/lib/python3.12/site-packages/packaging/requirements.py", line 37, in __init__
    raise InvalidRequirement(str(e)) from e
packaging.requirements.InvalidRequirement: Expected end or semicolon (after version specifier)
    flit-core === 3.9.0 --hash=sha256:7aada352fb0c7f5538c4fafeddf314d3a6a92ee8e2b1de70482329e42de70301
              ~~~~~~~~~~^

If I remove the --hash option, conda-lock succeeds.

Apparently conda is able to process the file: conda env create -f conda-linux-64.lock.yml works fine.

Conda Info

Output 
active environment : myenv 
    active env location : ###/myenv
            shell level : 1
       user config file : ###/.condarc
 populated config files : ###/.condarc
                          ###/.condarc
          conda version : 24.3.0
    conda-build version : not installed
         python version : 3.10.6.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=broadwell
                          __conda=24.3.0=0
                          __glibc=2.17=0
                          __linux=3.10.0=0
                          __unix=0=0
       base environment : ###/conda/latest  (read only)
      conda av data dir : ###/conda/latest/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : ###/conda/pkgs
       envs directories : ###/conda/envs
               platform : linux-64
             user-agent : conda/24.3.0 requests/2.31.0 CPython/3.10.6 Linux/3.10.0-1160.92.1.el7.x86_64 centos/7.9.2009 glibc/2.17 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8
                UID:GID : 13865:16016
             netrc file : ###/.netrc
           offline mode : False

Conda Config

Output 
==> ###/conda/latest/.condarc <==
channels:
  - conda-forge

==> /home/###/.condarc <==
envs_dirs:
  - ###/conda/envs
pkgs_dirs:
  - ###/conda/pkgs

Conda list

Output 
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
annotated-types           0.6.0              pyhd8ed1ab_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
aspell                    0.60.8          pl5321hcb278e6_1    conda-forge
brotli-python             1.1.0           py312h30efb56_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.28.1               hd590300_0    conda-forge
ca-certificates           2024.2.2             hbcca054_0    conda-forge
cachecontrol              0.14.0             pyhd8ed1ab_0    conda-forge
cachecontrol-with-filecache 0.14.0             pyhd8ed1ab_0    conda-forge
cachy                     0.3.0              pyhd8ed1ab_1    conda-forge
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py312hf06ca03_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
cleo                      2.1.0              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
click-default-group       1.2.4              pyhd8ed1ab_0    conda-forge
clikit                    0.6.2              pyhd8ed1ab_2    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda-lock                2.5.6              pyhd8ed1ab_0    conda-forge
crashtest                 0.4.1              pyhd8ed1ab_0    conda-forge
cryptography              42.0.5          py312h241aef2_0    conda-forge
curl                      8.7.1                hca28451_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
distlib                   0.3.8              pyhd8ed1ab_0    conda-forge
dulwich                   0.21.7          py312h98912ed_0    conda-forge
ensureconda               1.4.4              pyhd8ed1ab_0    conda-forge
expat                     2.6.2                h59595ed_0    conda-forge
filelock                  3.13.4             pyhd8ed1ab_0    conda-forge
gettext                   0.22.5               h59595ed_2    conda-forge
gettext-tools             0.22.5               h59595ed_2    conda-forge
git                       2.44.0          pl5321h709897a_0    conda-forge
gitdb                     4.0.11             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.43             pyhd8ed1ab_0    conda-forge
html5lib                  1.1                pyh9f0ad1d_0    conda-forge
idna                      3.7                pyhd8ed1ab_0    conda-forge
importlib-metadata        7.1.0              pyha770c72_0    conda-forge
importlib_metadata        7.1.0                hd8ed1ab_0    conda-forge
importlib_resources       6.4.0              pyhd8ed1ab_0    conda-forge
jaraco.classes            3.4.0              pyhd8ed1ab_0    conda-forge
jaraco.context            4.3.0              pyhd8ed1ab_0    conda-forge
jaraco.functools          4.0.0              pyhd8ed1ab_0    conda-forge
jeepney                   0.8.0              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.3              pyhd8ed1ab_0    conda-forge
keyring                   24.3.1          py312h7900ff3_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libasprintf               0.22.5               h661eb56_2    conda-forge
libasprintf-devel         0.22.5               h661eb56_2    conda-forge
libblas                   3.9.0           22_linux64_openblas    conda-forge
libcblas                  3.9.0           22_linux64_openblas    conda-forge
libcurl                   8.7.1                hca28451_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.6.2                h59595ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_5    conda-forge
libgettextpo              0.22.5               h59595ed_2    conda-forge
libgettextpo-devel        0.22.5               h59595ed_2    conda-forge
libgfortran-ng            13.2.0               h69a702a_5    conda-forge
libgfortran5              13.2.0               ha4646dd_5    conda-forge
libglib                   2.80.0               hf2295e7_5    conda-forge
libgomp                   13.2.0               h807b86a_5    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
liblapack                 3.9.0           22_linux64_openblas    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.27          pthreads_h413a1c8_0    conda-forge
libsqlite                 3.45.3               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
markupsafe                2.1.5           py312h98912ed_0    conda-forge
more-itertools            10.2.0             pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.7           py312h8572e83_0    conda-forge
ncurses                   6.4.20240210         h59595ed_0    conda-forge
numpy                     1.26.4          py312heda63a1_0    conda-forge
openssl                   3.2.1                hd590300_1    conda-forge
packaging                 24.0               pyhd8ed1ab_0    conda-forge
pastel                    0.2.1              pyhd8ed1ab_0    conda-forge
pcre2                     10.43                hcad00b1_0    conda-forge
perl                      5.32.1          7_hd590300_perl5    conda-forge
pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
pkginfo                   1.10.0             pyhd8ed1ab_0    conda-forge
platformdirs              4.2.0              pyhd8ed1ab_0    conda-forge
poetry                    1.8.2           linux_pyha804496_0    conda-forge
poetry-core               1.9.0              pyhd8ed1ab_0    conda-forge
poetry-plugin-export      1.7.1              pyhd8ed1ab_1    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pycparser                 2.22               pyhd8ed1ab_0    conda-forge
pydantic                  2.7.0              pyhd8ed1ab_0    conda-forge
pydantic-core             2.18.1          py312h4b3b743_0    conda-forge
pylev                     1.4.0              pyhd8ed1ab_0    conda-forge
pyproject_hooks           1.0.0              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.12.3          hab00c5b_0_cpython    conda-forge
python-build              1.2.1              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.19.1             pyhd8ed1ab_0    conda-forge
python-installer          0.7.0              pyhd8ed1ab_0    conda-forge
python_abi                3.12                    4_cp312    conda-forge
pyyaml                    6.0.1           py312h98912ed_1    conda-forge
rapidfuzz                 3.8.1           py312h30efb56_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
requests-toolbelt         1.0.0              pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.18.6          py312h98912ed_0    conda-forge
ruamel.yaml.clib          0.2.8           py312h98912ed_0    conda-forge
secretstorage             3.3.3           py312h7900ff3_2    conda-forge
setuptools                69.5.1             pyhd8ed1ab_0    conda-forge
shellingham               1.5.4              pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     5.0.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tomlkit                   0.12.4             pyha770c72_0    conda-forge
toolz                     0.12.1             pyhd8ed1ab_0    conda-forge
trove-classifiers         2024.4.10          pyhd8ed1ab_0    conda-forge
typing-extensions         4.11.0               hd8ed1ab_0    conda-forge
typing_extensions         4.11.0             pyha770c72_0    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
urllib3                   1.26.18            pyhd8ed1ab_0    conda-forge
virtualenv                20.25.3            pyhd8ed1ab_0    conda-forge
webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
wheel                     0.43.0             pyhd8ed1ab_1    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

Additional Context

We use that workflow to add new packages to a lockfile while keeping existing packages as constant as possible.

  1. Render environment from lockfile
  2. Add new package to environment file
  3. Create new lockfile from modified environment file
  4. Commit lockfile changes
@maresb
Copy link
Contributor

maresb commented Apr 19, 2024

Thanks so much @sfinkens for the clear and easily-reproducible report!

In summary, the generated conda-linux-64.lock.yml contains

  - pip:
    - flit-core === 3.9.0 --hash=sha256:7aada352fb0c7f5538c4fafeddf314d3a6a92ee8e2b1de70482329e42de70302

Conda-lock tries to parse this flit-core dependency line here, feeding it into the packaging.Requirements class, which doesn't support the --hash=... representation.

I'm wondering why pip's hash-checking mode is not part of the requirements specification defined in the dependency specifiers grammar. (That seems to be the reason it's not handled by the packaging.Requirements class.)

@sfinkens, would you be willing to look into how to resolve this and create a PR?

@sfinkens
Copy link
Contributor Author

@maresb Thanks for your quick reply! Sure, I'll give it a try. What strategy would you prefer? Just strip the hash or try to pass it through to where the lockfile is created?

@maresb
Copy link
Contributor

maresb commented Apr 19, 2024 •
edited
Loading

@sfinkens, wonderful, thanks so much!

I'd prefer to pass through the lockfile hashes, and at first glance it doesn't seem like they need to travel very far. But if that proves too difficult we could also emit a clear warning and strip it.

@sfinkens sfinkens mentioned this issue Apr 26, 2024
Merged
@sfinkens
Copy link
Contributor Author

sfinkens commented Apr 29, 2024 •
edited
Loading

@maresb While debugging my example I noticed that our workflow might be too complicated. Is it correct, that conda-lock uses an existing lockfile as constraint for the next solution? For example

conda-lock -f environment.yml
# A couple weeks later: Add a new package to environment.yml
conda-lock -f environment.yml  # does this use conda-lock.yml as constraint?

I tried with a simple case and that seems to work. If that's the case, I think it would be nice to mention it in the documentation.

@maresb
Copy link
Contributor

maresb commented Apr 29, 2024

If there is an existing lockfile, it is indeed used when regenerating.

Personally, I agree that this behavior is really confusing, and I dislike it, but I also am reluctant to change it since it may break existing workflows. I'm in the habit of always deleting conda-lock.yml before relocking.

I'd be very happy to accept a documentation update to make this more clear.

@sfinkens
Copy link
Contributor Author

Thanks for the feedback! I'll prepare a PR for the documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add support for pip hash checking sfinkens/conda-lock
2 participants
@sfinkens @maresb