Problem with lacking hashes for pipenv with custom index url #4885
Comments
|
I agree with this report and was looking for someone with the same report -- our lock file is 75% lighter after version 2021.5.29. I am currently diffing the code base to see if I can find any hints at what could be going on. |
|
However @HenrikPoulsen, I am still able to reproduce the issue without having the [[source]] section in the Pipfile so I think that is maybe unrelated. Unfortunately my example is a private Pipfile that I cannot share, but it seems to depend on the versions of things in the Pipfile. For example, after modifying just the version of 1/65 requirements and adding another requirement to the Pipfile, the lock file goes from missing 75% of requirements to appearing to be complete. |
It does seem to rely upon running a custom index. I'm using devpi for that here.
It does seem to rely upon running a custom index. I'm using devpi for that here.
|
We're running into this as well. I've put together a repro here: https://github.com/jfly/pipenv-issue-4885-repro. Unlike @matteius but like @HenrikPoulsen, I needed to introduce an alternative Python package index to trigger the issue. So maybe @matteius is running into something different, or there are other ways to get this issue to rear its head. I'll try to get a clone of pipenv running and see if I can |
It does seem to rely upon running a custom index. I'm using devpi for that here.
|
So I bisected, and here's what I found:
That corresponds to this massive PR from @frostming: #4759, which definitely sounds suspicious. I'm not sure how much further I'll be able to take this investigation. |
|
Ok, I've dug enough to confirm that 5ef5a59 is definitely where this broke. Previous, working behavior (I'm looking at d0fa43a):
New, broken behavior (I'm looking at 3ab4763 here):
Unfortunately, this seems to be one of those situations where understanding why it used to work and why it doesn't work now doesn't seem to be enough information to fix the problem. I don't know why we moved away from piptools, but presumably that was for a good reason. Maybe the right fix is to change up this code to find all candidates, not just ones that could work on the current platform? @frostming, I could really use some help here. Thanks! |
This should be wrong, care to submit a fix for it? @jfly Thanks. |
|
@frostming, I can give that a shot. I could use a little help getting started, though. My initial thought is to add a boolean parameter to |
|
Oh wait, I just realized I'm talking about changing vendored pip code. Is that an ok? Or do we prefer not to change that code? |
|
@jfly This other issue #4866 was specifically what was causing the missing hashes in my case; Based on your research so far do you think this could be related to what you have found? #4866 (comment) |
|
@matteius that sounds different. The bug I tracked down is definitely specific to when you're using a custom python package server. |
2022.1.8 (2022-01-08) ===================== Bug Fixes --------- - Remove the extra parentheses around the venv prompt. `#4877 <https://github.com/pypa/pipenv/issues/4877>`_ - Fix a bug of installation fails when extra index url is given. `#4881 <https://github.com/pypa/pipenv/issues/4881>`_ - Fix regression where lockfiles would only include the hashes for releases for the platform generating the lockfile `#4885 <https://github.com/pypa/pipenv/issues/4885>`_ - Fix the index parsing to reject illegal requirements.txt. `#4899 <https://github.com/pypa/pipenv/issues/4899>`_ 2021.11.23 (2021-11-23) ======================= Bug Fixes --------- - Update ``charset-normalizer`` from ``2.0.3`` to ``2.0.7``, this fixes an import error on Python 3.6. `#4865 <https://github.com/pypa/pipenv/issues/4865>`_ - Fix a bug of deleting a virtualenv that is not managed by Pipenv. `#4867 <https://github.com/pypa/pipenv/issues/4867>`_ - Fix a bug that source is not added to ``Pipfile`` when index url is given with ``pipenv install``. `#4873 <https://github.com/pypa/pipenv/issues/4873>`_ 2021.11.15 (2021-11-15) ======================= Bug Fixes --------- - Return an empty dict when ``PIPENV_DONT_LOAD_ENV`` is set. `#4851 <https://github.com/pypa/pipenv/issues/4851>`_ - Don't use ``sys.executable`` when inside an activated venv. `#4852 <https://github.com/pypa/pipenv/issues/4852>`_ Vendored Libraries ------------------ - Drop the vendored ``jinja2`` dependency as it is not needed any more. `#4858 <https://github.com/pypa/pipenv/issues/4858>`_ - Update ``click`` from ``8.0.1`` to ``8.0.3``, to fix a problem with bash completion. `#4860 <https://github.com/pypa/pipenv/issues/4860>`_ - Drop unused vendor ``chardet``. `#4862 <https://github.com/pypa/pipenv/issues/4862>`_ Improved Documentation ---------------------- - Fix the documentation to reflect the fact that special characters must be percent-encoded in the URL. `#4856 <https://github.com/pypa/pipenv/issues/4856>`_ 2021.11.9 (2021-11-09) ====================== Features & Improvements ----------------------- - Replace ``click-completion`` with ``click``'s own completion implementation. `#4786 <https://github.com/pypa/pipenv/issues/4786>`_ Bug Fixes --------- - Fix a bug that ``pipenv run`` doesn't set environment variables correctly. `#4831 <https://github.com/pypa/pipenv/issues/4831>`_ - Fix a bug that certifi can't be loaded within ``notpip``'s vendor library. This makes several objects of ``pip`` fail to be imported. `#4833 <https://github.com/pypa/pipenv/issues/4833>`_ - Fix a bug that ``3.10.0`` can be found be python finder. `#4837 <https://github.com/pypa/pipenv/issues/4837>`_ Vendored Libraries ------------------ - Update ``pythonfinder`` from ``1.2.8`` to ``1.2.9``. `#4837 <https://github.com/pypa/pipenv/issues/4837>`_ 2021.11.5.post0 (2021-11-05) ============================ Bug Fixes --------- - Fix a regression that ``pipenv shell`` fails to start a subshell. `#4828 <https://github.com/pypa/pipenv/issues/4828>`_ - Fix a regression that ``pip_shims`` object isn't imported correctly. `#4829 <https://github.com/pypa/pipenv/issues/4829>`_ 2021.11.5 (2021-11-05) ====================== Features & Improvements ----------------------- - Avoid sharing states but create project objects on demand. So that most integration test cases are able to switch to a in-process execution method. `#4757 <https://github.com/pypa/pipenv/issues/4757>`_ - Shell-quote ``pip`` commands when logging. `#4760 <https://github.com/pypa/pipenv/issues/4760>`_ Bug Fixes --------- - Ignore empty .venv in rood dir and create project name base virtual environment `#4790 <https://github.com/pypa/pipenv/issues/4790>`_ Vendored Libraries ------------------ - Update vendored dependencies - ``attrs`` from ``20.3.0`` to ``21.2.0`` - ``cerberus`` from ``1.3.2`` to ``1.3.4`` - ``certifi`` from ``2020.11.8`` to ``2021.5.30`` - ``chardet`` from ``3.0.4`` to ``4.0.0`` - ``click`` from ``7.1.2`` to ``8.0.1`` - ``distlib`` from ``0.3.1`` to ``0.3.2`` - ``idna`` from ``2.10`` to ``3.2`` - ``importlib-metadata`` from ``2.0.0`` to ``4.6.1`` - ``importlib-resources`` from ``3.3.0`` to ``5.2.0`` - ``jinja2`` from ``2.11.2`` to ``3.0.1`` - ``markupsafe`` from ``1.1.1`` to ``2.0.1`` - ``more-itertools`` from ``5.0.0`` to ``8.8.0`` - ``packaging`` from ``20.8`` to ``21.0`` - ``pep517`` from ``0.9.1`` to ``0.11.0`` - ``pipdeptree`` from ``1.0.0`` to ``2.0.0`` - ``ptyprocess`` from ``0.6.0`` to ``0.7.0`` - ``python-dateutil`` from ``2.8.1`` to ``2.8.2`` - ``python-dotenv`` from ``0.15.0`` to ``0.19.0`` - ``pythonfinder`` from ``1.2.5`` to ``1.2.8`` - ``requests`` from ``2.25.0`` to ``2.26.0`` - ``shellingham`` from ``1.3.2`` to ``1.4.0`` - ``six`` from ``1.15.0`` to ``1.16.0`` - ``tomlkit`` from ``0.7.0`` to ``0.7.2`` - ``urllib3`` from ``1.26.1`` to ``1.26.6`` - ``zipp`` from ``1.2.0`` to ``3.5.0`` Add new vendored dependencies - ``charset-normalizer 2.0.3`` - ``termcolor 1.1.0`` - ``tomli 1.1.0`` - ``wheel 0.36.2`` `#4747 <https://github.com/pypa/pipenv/issues/4747>`_ - Drop the dependencies for Python 2.7 compatibility purpose. `#4751 <https://github.com/pypa/pipenv/issues/4751>`_ - Switch the dependency resolver from ``pip-tools`` to `pip`. Update vendor libraries: - Update ``requirementslib`` from ``1.5.16`` to ``1.6.1`` - Update ``pip-shims`` from ``0.5.6`` to ``0.6.0`` - New vendor ``platformdirs 2.4.0`` `#4759 <https://github.com/pypa/pipenv/issues/4759>`_ Improved Documentation ---------------------- - remove prefixes on install commands for easy copy/pasting `#4792 <https://github.com/pypa/pipenv/issues/4792>`_ - Officially drop support for Python 2.7 and Python 3.5. `#4261 <https://github.com/pypa/pipenv/issues/4261>`_ 2021.5.29 (2021-05-29) ====================== Bug Fixes --------- - Fix a bug where passing --skip-lock when PIPFILE has no [SOURCE] section throws the error: "tomlkit.exceptions.NonExistentKey: 'Key "source" does not exist.'" `#4141 <https://github.com/pypa/pipenv/issues/4141>`_ - Fix bug where environment wouldn't activate in paths containing & and $ symbols `#4538 <https://github.com/pypa/pipenv/issues/4538>`_ - Fix a bug that ``importlib-metadata`` from the project's dependencies conflicts with that from ``pipenv``'s. `#4549 <https://github.com/pypa/pipenv/issues/4549>`_ - Fix a bug where ``pep508checker.py`` did not expect double-digit Python minor versions (e.g. "3.10"). `#4602 <https://github.com/pypa/pipenv/issues/4602>`_ - Fix bug where environment wouldn't activate in paths containing () and [] symbols `#4615 <https://github.com/pypa/pipenv/issues/4615>`_ - Fix bug preventing use of pipenv lock --pre `#4642 <https://github.com/pypa/pipenv/issues/4642>`_ Vendored Libraries ------------------ - Update ``packaging`` from ``20.4`` to ``20.8``. `#4591 <https://github.com/pypa/pipenv/issues/4591>`_ 2020.11.15 (2020-11-15) ======================= Features & Improvements ----------------------- - Support expanding environment variables in requirement URLs. `#3516 <https://github.com/pypa/pipenv/issues/3516>`_ - Show warning message when a dependency is skipped in locking due to the mismatch of its markers. `#4346 <https://github.com/pypa/pipenv/issues/4346>`_ Bug Fixes --------- - Fix a bug that executable scripts with leading backslash can't be executed via ``pipenv run``. `#4368 <https://github.com/pypa/pipenv/issues/4368>`_ - Fix a bug that VCS dependencies always satisfy even if the ref has changed. `#4387 <https://github.com/pypa/pipenv/issues/4387>`_ - Restrict the acceptable hash type to SHA256 only. `#4517 <https://github.com/pypa/pipenv/issues/4517>`_ - Fix the output of ``pipenv scripts`` under Windows platform. `#4523 <https://github.com/pypa/pipenv/issues/4523>`_ - Fix a bug that the resolver takes wrong section to validate constraints. `#4527 <https://github.com/pypa/pipenv/issues/4527>`_ Vendored Libraries ------------------ - Update vendored dependencies: - ``colorama`` from ``0.4.3`` to ``0.4.4`` - ``python-dotenv`` from ``0.10.3`` to ``0.15.0`` - ``first`` from ``2.0.1`` to ``2.0.2`` - ``iso8601`` from ``0.1.12`` to ``0.1.13`` - ``parse`` from ``1.15.0`` to ``1.18.0`` - ``pipdeptree`` from ``0.13.2`` to ``1.0.0`` - ``requests`` from ``2.23.0`` to ``2.25.0`` - ``idna`` from ``2.9`` to ``2.10`` - ``urllib3`` from ``1.25.9`` to ``1.26.1`` - ``certifi`` from ``2020.4.5.1`` to ``2020.11.8`` - ``requirementslib`` from ``1.5.15`` to ``1.5.16`` - ``attrs`` from ``19.3.0`` to ``20.3.0`` - ``distlib`` from ``0.3.0`` to ``0.3.1`` - ``packaging`` from ``20.3`` to ``20.4`` - ``six`` from ``1.14.0`` to ``1.15.0`` - ``semver`` from ``2.9.0`` to ``2.13.0`` - ``toml`` from ``0.10.1`` to ``0.10.2`` - ``cached-property`` from ``1.5.1`` to ``1.5.2`` - ``yaspin`` from ``0.14.3`` to ``1.2.0`` - ``resolvelib`` from ``0.3.0`` to ``0.5.2`` - ``pep517`` from ``0.8.2`` to ``0.9.1`` - ``zipp`` from ``0.6.0`` to ``1.2.0`` - ``importlib-metadata`` from ``1.6.0`` to ``2.0.0`` - ``importlib-resources`` from ``1.5.0`` to ``3.3.0`` `#4533 <https://github.com/pypa/pipenv/issues/4533>`_ Improved Documentation ---------------------- - Fix suggested pyenv setup to avoid using shimmed interpreter `#4534 <https://github.com/pypa/pipenv/issues/4534>`_ 2020.11.4 (2020-11-04) ====================== Features & Improvements ----------------------- - Add a new command ``pipenv scripts`` to display shortcuts from Pipfile. `#3686 <https://github.com/pypa/pipenv/issues/3686>`_ - Retrieve package file hash from URL to accelerate the locking process. `#3827 <https://github.com/pypa/pipenv/issues/3827>`_ - Add the missing ``--system`` option to ``pipenv sync``. `#4441 <https://github.com/pypa/pipenv/issues/4441>`_ - Add a new option pair ``--header/--no-header`` to ``pipenv lock`` command, which adds a header to the generated requirements.txt `#4443 <https://github.com/pypa/pipenv/issues/4443>`_ Bug Fixes --------- - Fix a bug that percent encoded characters will be unquoted incorrectly in the file URL. `#4089 <https://github.com/pypa/pipenv/issues/4089>`_ - Fix a bug where setting PIPENV_PYTHON to file path breaks environment name `#4225 <https://github.com/pypa/pipenv/issues/4225>`_ - Fix a bug that paths are not normalized before comparison. `#4330 <https://github.com/pypa/pipenv/issues/4330>`_ - Handle Python major and minor versions correctly in Pipfile creation. `#4379 <https://github.com/pypa/pipenv/issues/4379>`_ - Fix a bug that non-wheel file requirements can be resolved successfully. `#4386 <https://github.com/pypa/pipenv/issues/4386>`_ - Fix a bug that ``pexept.exceptions.TIMEOUT`` is not caught correctly because of the wrong import path. `#4424 <https://github.com/pypa/pipenv/issues/4424>`_ - Fix a bug that compound TOML table is not parsed correctly. `#4433 <https://github.com/pypa/pipenv/issues/4433>`_ - Fix a bug that invalid Python paths from Windows registry break ``pipenv install``. `#4436 <https://github.com/pypa/pipenv/issues/4436>`_ - Fix a bug that function calls in ``setup.py`` can't be parsed rightly. `#4446 <https://github.com/pypa/pipenv/issues/4446>`_ - Fix a bug that dist-info inside ``venv`` directory will be mistaken as the editable package's metadata. `#4480 <https://github.com/pypa/pipenv/issues/4480>`_ - Make the order of hashes in resolution result stable. `#4513 <https://github.com/pypa/pipenv/issues/4513>`_ Vendored Libraries ------------------ - Update ``tomlkit`` from ``0.5.11`` to ``0.7.0``. `#4433 <https://github.com/pypa/pipenv/issues/4433>`_ - Update ``requirementslib`` from ``1.5.13`` to ``1.5.14``. `#4480 <https://github.com/pypa/pipenv/issues/4480>`_ Improved Documentation ---------------------- - Discourage homebrew installation in installation guides. `#4013 <https://github.com/pypa/pipenv/issues/4013>`_
Issue description
Since some version after pipenv 2021.5.29 it seems the lockfile behavior has changed regarding the hashes that get written to the file.
I use Renovate to create PRs for updated dependencies. And since a couple of weeks the repo started to get lockfiles produced that were missing windows hashes for dependencies.
Specifically it updates from aiohttp 3.8.0 to 3.8.1 and it ends up only including two hashes for it in the lockfile after it is done.
When CI then tries to run the windows unit tests on that PR it fails, due to the hashes it is looking for is not present in the lockfile.
I have noticed that this only happens if I have a custom index-url in the Pipfile, which in my case points to an artifactory instance we use.
Expected result
Lockfile should be updated with all available hashes so crossplatform
pipenv installis successfulActual result
I get this error message on windows after it pip installs using the lockfile that renovate has created:
It only includes 2 hashes instead of the 70+ as on the older pipenv version
This is the verbose output of when it doesn't work:
Steps to replicate
pipenv --supportbelow here, modify the index-url to something that works for youdocker run -it -v /folder/with/pipfile:/data --entrypoint /bin/bash renovate/python:3.9use whatever you preferpip install pipenvpip installPipfile.lockonly has a couple of values in itObserve that I was unable to reproduce this without the
[[source]]section in the Pipfile$ pipenv --support
Pipenv version:
'2021.11.23'Pipenv location:
'/home/ubuntu/.local/lib/python3.9/site-packages/pipenv'Python location:
'/usr/local/python/3.9.9/bin/python3.9'Python installations found:
3.9.9:/usr/local/python/3.9.9/bin/python3.93.9.9:/usr/local/python/3.9.9/bin/python33.9.9:/usr/local/python/3.9.9/bin/python3.9.9:/usr/local/bin/pythonPEP 508 Information:
System environment variables:
USER_IDHOSTNAMEPWDUSER_NAMEHOMEUSER_HOMELANGLS_COLORSTERMSHLVLBASH_ENVLC_ALLPATHDEBIAN_FRONTENDBUILDPACKOLDPWD_PIP_SHIMS_BASE_MODULEPIP_DISABLE_PIP_VERSION_CHECKPYTHONDONTWRITEBYTECODEPIP_PYTHON_PATHPYTHONFINDER_IGNORE_UNSUPPORTEDPipenv–specific environment variables:
Debug–specific environment variables:
PATH:/home/ubuntu/.local/bin:/usr/local/python/3.9.9/bin:/home/ubuntu/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binLANG:C.UTF-8PWD:/dataContents of
Pipfile('/data/Pipfile'):Contents of
Pipfile.lock('/data/Pipfile.lock'):{ "_meta": { "hash": { "sha256": "124cf509d4d27ce063e7a4f1fe5faf9892b0896c8e84f4207b11663cc780b28e" }, "pipfile-spec": 6, "requires": {}, "sources": [ { "name": "pypi", "url": "https//someartifactory/api/pypi/pypi/simple", "verify_ssl": true } ] }, "default": { "aiohttp": { "hashes": [ "sha256:d15367ce87c8e9e09b0f989bfd72dc641bcd04ba091c68cd305312d00962addd", "sha256:fc5471e1a54de15ef71c1bc6ebe80d4dc681ea600e68bfd1cbce40427f0b7578" ], "index": "pypi", "version": "==3.8.1" }, "aiosignal": { "hashes": [ "sha256:26e62109036cd181df6e6ad646f91f0dcfd05fe16d0cb924138ff2ab75d64e3a", "sha256:78ed67db6c7b7ced4f98e495e572106d5c432a93e1ddd1bf475e1dc05f5b7df2" ], "markers": "python_version >= '3.6'", "version": "==1.2.0" }, "async-timeout": { "hashes": [ "sha256:a22c0b311af23337eb05fcf05a8b51c3ea53729d46fb5460af62bee033cec690", "sha256:b930cb161a39042f9222f6efb7301399c87eeab394727ec5437924a36d6eef51" ], "markers": "python_version >= '3.6'", "version": "==4.0.1" }, "attrs": { "hashes": [ "sha256:149e90d6d8ac20db7a955ad60cf0e6881a3f20d37096140088356da6c716b0b1", "sha256:ef6aaac3ca6cd92904cdd0d83f629a15f18053ec84e6432106f7a4d04ae4f5fb" ], "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'", "version": "==21.2.0" }, "charset-normalizer": { "hashes": [ "sha256:735e240d9a8506778cd7a453d97e817e536bb1fc29f4f6961ce297b9c7a917b0", "sha256:83fcdeb225499d6344c8f7f34684c2981270beacc32ede2e669e94f7fa544405" ], "markers": "python_full_version >= '3.5.0'", "version": "==2.0.8" }, "frozenlist": { "hashes": [ "sha256:1fef737fd1388f9b93bba8808c5f63058113c10f4e3c0763ced68431773f72f9", "sha256:68201be60ac56aff972dc18085800b6ee07973c49103a8aba669dee3d71079de" ], "markers": "python_version >= '3.6'", "version": "==1.2.0" }, "idna": { "hashes": [ "sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff", "sha256:9d643ff0a55b762d5cdb124b8eaa99c66322e2157b69160bc32796e824360e6d" ], "markers": "python_full_version >= '3.5.0'", "version": "==3.3" }, "multidict": { "hashes": [ "sha256:0dd1c93edb444b33ba2274b66f63def8a327d607c6c790772f448a53b6ea59ce", "sha256:93de39267c4c676c9ebb2057e98a8138bade0d806aad4d864322eee0803140a0" ], "markers": "python_version >= '3.6'", "version": "==5.2.0" }, "typing-extensions": { "hashes": [ "sha256:2cdf80e4e04866a9b3689a51869016d36db0814d84b8d8a568d22781d45d27ed", "sha256:829704698b22e13ec9eaf959122315eabb370b0884400e9818334d8b677023d9" ], "markers": "python_version >= '3.6'", "version": "==4.0.0" }, "yarl": { "hashes": [ "sha256:45399b46d60c253327a460e99856752009fcee5f5d3c80b2f7c0cae1c38d56dd", "sha256:f64394bd7ceef1237cc604b5a89bf748c95982a84bcd3c4bbeb40f685c810794" ], "markers": "python_version >= '3.6'", "version": "==1.7.2" } }, "develop": {} }The text was updated successfully, but these errors were encountered: