pip install fails with ValueError: too many values to unpack
#6165
Comments
Same issue when installing (when invoking using tox):
as others have verified, this chokes on files that have commas in their name. |
|
It looks like this code change was introduced in this merge commit 7696e7e (specifically b1c33bd ) /cc @mkurnikov |
|
I grabbed an example of some of the supposed 3-tuples this code expects: Less-than correct: |
|
@rcramblit Can you determine who or what generated the files causing the problem? |
|
@cjerdonek Looks like these particular ones are coming from: https://github.com/statsmodels/statsmodels/tree/v0.6.1/statsmodels/datasets/tests Filenames with commas in them.. |
|
And how about who / what generated the csv ( |
|
Not sure I get what you're asking - Aren't these created by pip when a package is installed? When I look at the RECORD file for statsmodels ( Edit - That is from an installation using |
|
I'm not very knowledgeable on this aspect, but for example, here's a similar issue involving commas in the filename where the issue seems to be with an original RECORD file, and the file wasn't generated by pip but rather by the wheel project: #6103 So one possibility of what might be happening is that pip reads an existing RECORD file (possibly externally generated) and generates a new one, and what you've included above is the new one rather than the original one. The change seems to be that previously, pip was "loose" in what RECORD files it could read (which is why 18.1 was able to install the project you mention), whereas now it became more strict, so you're seeing a problem with a RECORD file that previously wasn't complying. I was trying to trace where the original RECORD file came from. You mentioned "pre-built" packages in your original post. |
|
Ah I see. I think you hit it on the nose. I see in the build logs |
This comment has been minimized.
This comment has been minimized.
cjerdonek
added
C: wheel
That's on me. ISTM that I didn't review that PR properly. I'll look into this and file a bugfix PR. |
|
I'm hitting this bug when I try to |
cjerdonek
added
type: bug
|
Okay yeah, this is a problem on the side of wheels being built and pip 19.0 caring about getting more strict as @cjerdonek noted; not the code change pertaining to the type annotations PR. (yay! I did review the code properly!) pip 18.1 was a lot kinder, but would generate invalid RECORD files and fail to properly handle uninstalls. pip/src/pip/_internal/wheel.py Lines 504 to 508 in 6af9de9 So, this isn't really a bug introduced in pip 19.0, so I'll drop the "bug". That said, @pypa/pip-committers do we want to be kind and just print a warning or be strict and refuse to install? We can (and should) add an error message here regardless; instead of a bad traceback. :) wheel issue: pypa/wheel#280 |
|
My inclination at this point would be to return to the previous (non-strict) behavior and discuss doing something more or different in a slower time frame. Some of us had discussed being strict with regard to RECORD files in a different, related issue, and we decided against it for the time being IIRC (see #5868 and #5913, for example). If there's an easy way to log a warning like by doing a length check, perhaps that would be okay. (I'm also not sure that the RECORD winds up invalid because I think pip generates a new, correct one. The commenter said above that the resulting file looked okay.) By the way, you might have misunderstood my comment about the type annotations. What I was saying is that the code change that caused this (refactoring out a |
Ah right, that makes more sense. :) |
IIUC, wheels generated with
Fair enough. Coming from #5913 (comment) -> I think this falls in the second category. That said, a bodge I have in mind is |
Ok, I spoke wrong -- it's when pip's installing with that it rewrites |
Certbot would really, really appreciate this, as the change breaks certbot for all of our certbot-auto users. Can we get the change reverted now, and fixed fully later? Otherwise we'll have to patch and release a point version of Certbot ASAP, before people's certs start expiring. Some of our test files have commas in filenames. We plan to address this in a future release, but it seems safer all around to revert the breaking change and then fix things carefully in tandem. |
|
FYI, I added a PR for this here: #6191 |
|
@pradyunsg will there be a 19.0.2 including the fix for this? |
|
@sbidoul Yes, see here: #6106 (comment) |
|
I had this issue and updating my |
1726: Update python:3.7-slim Docker digest to 8a125f6 r=peterbe a=renovate[bot] This PR contains the following updates: | Package | Update | Change | |---|---|---| | python | digest | `3.7-slim` -> `8a125f6` | 1732: Scheduled weekly dependency update for week 06 r=peterbe a=pyup-bot ### Update [botocore](https://pypi.org/project/botocore) from **1.12.86** to **1.12.91**. <details> <summary>Changelog</summary> ### 1.12.91 ``` ======= * api-change:``discovery``: Update discovery client to latest version * api-change:``ecs``: Update ecs client to latest version * api-change:``dlm``: Update dlm client to latest version ``` ### 1.12.90 ``` ======= * api-change:``es``: Update es client to latest version * api-change:``medialive``: Update medialive client to latest version * api-change:``gamelift``: Update gamelift client to latest version * api-change:``robomaker``: Update robomaker client to latest version ``` ### 1.12.89 ``` ======= * api-change:``ec2``: Update ec2 client to latest version * api-change:``fsx``: Update fsx client to latest version ``` ### 1.12.88 ``` ======= * api-change:``shield``: Update shield client to latest version * api-change:``servicecatalog``: Update servicecatalog client to latest version * api-change:``ec2``: Update ec2 client to latest version ``` ### 1.12.87 ``` ======= * api-change:``ecs``: Update ecs client to latest version * api-change:``application-autoscaling``: Update application-autoscaling client to latest version * api-change:``workspaces``: Update workspaces client to latest version * api-change:``codecommit``: Update codecommit client to latest version ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/botocore - Changelog: https://pyup.io/changelogs/botocore/ - Repo: https://github.com/boto/botocore </details> ### Update [python-dateutil](https://pypi.org/project/python-dateutil) from **2.7.5** to **2.8.0**. <details> <summary>Changelog</summary> ### 2.8.0 ``` ========================== Data updates ------------ - Updated tzdata version to to 2018i. Features -------- - Added support for ``EXDATE`` parameters when parsing ``rrule`` strings. Reported by mlorant (gh issue 410), fixed by nicoe (gh pr 859). - Added support for sub-minute time zone offsets in Python 3.6+. Fixed by cssherry (gh issue 582, pr 763) - Switched the ``tzoffset``, ``tzstr`` and ``gettz`` caches over to using weak references, so that the cache expires when no other references to the original ``tzinfo`` objects exist. This cache-expiry behavior is not guaranteed in the public interface and may change in the future. To improve performance in the case where transient references to the same time zones are repeatedly created but no strong reference is continuously held, a smaller "strong value" cache was also added. Weak value cache implemented by cs-cordero (gh pr 672, 801), strong cache added by Gökçen Nurlu (gh issue 691, gh pr 761) Bugfixes -------- - Added time zone inference when initializing an ``rrule`` with a specified ``UNTIL`` but without an explicitly specified ``DTSTART``; the time zone of the generated ``DTSTART`` will now be taken from the ``UNTIL`` rule. Reported by href (gh issue 652). Fixed by absreim (gh pr 693). - Fixed an issue where ``parser.parse`` would raise ``Decimal``-specific errors instead of a standard ``ValueError`` if certain malformed values were parsed (e.g. ``NaN`` or infinite values). Reported and fixed by amureki (gh issue 662, gh pr 679). - Fixed issue in ``parser`` where a ``tzinfos`` call explicitly returning ``None`` would throw a ``ValueError``. Fixed by parsethis (gh issue 661, gh pr 681) - Fixed incorrect parsing of certain dates earlier than 100 AD when repesented in the form "%B.%Y.%d", e.g. "December.0031.30". (gh issue 687, pr 700) - Add support for ISO 8601 times with comma as the decimal separator in the ``dateutil.parser.isoparse`` function. (gh pr 721) - Changed handling of ``T24:00`` to be compliant with the standard. ``T24:00`` now represents midnight on the *following* day. Fixed by cheukting (gh issue 658, gh pr 751) - Fixed an issue where ``isoparser.parse_isotime`` was unable to handle the ``24:00`` variant representation of midnight. (gh pr 773) - Added support for more than 6 fractional digits in `isoparse`. Reported and fixed by jayschwa (gh issue 786, gh pr 787). - Added 'z' (lower case Z) as valid UTC time zone in isoparser. Reported by cjgibson (gh issue 820). Fixed by Cheukting (gh pr 822) - Fixed a bug with base offset changes during DST in ``tzfile``, and refactored the way base offset changes are detected. Originally reported on StackOverflow by MartinThoma. (gh issue 812, gh pr 810) - Fixed error condition in ``tz.gettz`` when a non-ASCII timezone is passed on Windows in Python 2.7. (gh issue 802, pr 861) - Improved performance and inspection properties of ``tzname`` methods. (gh pr 811) - Removed unnecessary binary_type compatibility shims. Added by jdufresne (gh pr 817) - Changed ``python setup.py test`` to print an error to ``stderr`` and exit with 1 instead of 0. Reported and fixed by hroncok (gh pr 814) - Added a ``pyproject.toml`` file with build requirements and an explicitly specified build backend. (gh issue 736, gh prs 746, 863) Documentation changes --------------------- - Added documentation for the ``rrule.rrulestr`` function. Fixed by prdickson (gh issue 623, gh pr 762) - Added documentation for ``dateutil.tz.gettz``. Fixed by weatherpattern (gh issue 647, gh pr 704) - Add documentation for the ``dateutil.tz.win`` module and mocked out certain Windows-specific modules so that autodoc can still be run on non-Windows systems. (gh issue 442, pr 715) - Added changelog to documentation. (gh issue 692, gh pr 707) - Changed order of keywords in the ``rrule`` docstring. Reported and fixed by rmahajan14 (gh issue 686, gh pr 695). - Improved documentation on the use of ``until`` and ``count`` parameters in ``rrule``. Fixed by lucaferocino (gh pr 755). - Added an example of how to use a custom ``parserinfo`` subclass to parse non-standard datetime formats in the examples documentation for ``parser``. Added by prdickson (gh 753) - Added doctest examples to ``tzfile`` documentation. Patch by weatherpattern (gh pr 671) - Updated the documentation for ``relativedelta``'s ``weekday`` arguments. Fixed by kvn219 huangy22 and ElliotJH (gh pr 673) - Improved explanation of the order that ``relativedelta`` components are applied in. Fixed by kvn219 huangy22 and ElliotJH (gh pr 673) - Expanded the description and examples in the ``relativedelta`` class. Contributed by andrewcbennett (gh pr 759) - Improved the contributing documentation to clarify where to put new changelog files. Contributed by andrewcbennett (gh pr 757) - Fixed a broken doctest in the ``relativedelta`` module. Fixed by nherriot (gh pr 758). - Changed the default theme to ``sphinx_rtd_theme``, and changed the sphinx configuration accordingly. (gh pr 707) - Reorganized ``dateutil.tz`` documentation and fixed issue with the ``dateutil.tz`` docstring. (gh pr 714) - Cleaned up malformed RST in the ``tz`` documentation. (gh issue 702, gh pr 706) - Corrected link syntax and updated URL to https for ISO year week number notation in ``relativedelta`` examples. (gh issue 670, pr 711) Misc ---- - GH 674, GH 688, GH 699, GH 720, GH 723, GH 726, GH 727, GH 740, GH 750, GH 760, GH 767, GH 772, GH 773, GH 780, GH 784, GH 785, GH 791, GH 799, GH 813, GH 836, GH 839, GH 857 ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/python-dateutil - Changelog: https://pyup.io/changelogs/python-dateutil/ - Docs: https://dateutil.readthedocs.io </details> ### Update [s3transfer](https://pypi.org/project/s3transfer) from **0.1.13** to **0.2.0**. <details> <summary>Changelog</summary> ### 0.2.0 ``` ===== * feature:``ProcessPoolDownloader``: Add ``ProcessPoolDownloader`` class to speed up download throughput by using processes instead of threads. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/s3transfer - Changelog: https://pyup.io/changelogs/s3transfer/ - Repo: https://github.com/boto/s3transfer </details> ### Update [setuptools](https://pypi.org/project/setuptools) from **40.7.3** to **40.8.0**. <details> <summary>Changelog</summary> ### 40.8.0 ``` ------- * 1652: Added the ``build_meta:__legacy__`` backend, a "compatibility mode" PEP 517 backend that can be used as the default when ``build-backend`` is left unspecified in ``pyproject.toml``. * 1635: Resource paths are passed to ``pkg_resources.resource_string`` and similar no longer accept paths that traverse parents, that begin with a leading ``/``. Violations of this expectation raise DeprecationWarnings and will become errors. Additionally, any paths that are absolute on Windows are strictly disallowed and will raise ValueErrors. * 1536: ``setuptools`` will now automatically include licenses if ``setup.cfg`` contains a ``license_file`` attribute, unless this file is manually excluded inside ``MANIFEST.in``. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/setuptools - Changelog: https://pyup.io/changelogs/setuptools/ - Repo: https://github.com/pypa/setuptools </details> ### Update [boto3](https://pypi.org/project/boto3) from **1.9.86** to **1.9.91**. <details> <summary>Changelog</summary> ### 1.9.91 ``` ====== * api-change:``discovery``: [``botocore``] Update discovery client to latest version * api-change:``ecs``: [``botocore``] Update ecs client to latest version * api-change:``dlm``: [``botocore``] Update dlm client to latest version ``` ### 1.9.90 ``` ====== * api-change:``es``: [``botocore``] Update es client to latest version * api-change:``medialive``: [``botocore``] Update medialive client to latest version * api-change:``gamelift``: [``botocore``] Update gamelift client to latest version * api-change:``robomaker``: [``botocore``] Update robomaker client to latest version ``` ### 1.9.89 ``` ====== * api-change:``ec2``: [``botocore``] Update ec2 client to latest version * api-change:``fsx``: [``botocore``] Update fsx client to latest version ``` ### 1.9.88 ``` ====== * api-change:``shield``: [``botocore``] Update shield client to latest version * api-change:``servicecatalog``: [``botocore``] Update servicecatalog client to latest version * api-change:``ec2``: [``botocore``] Update ec2 client to latest version ``` ### 1.9.87 ``` ====== * api-change:``ecs``: [``botocore``] Update ecs client to latest version * api-change:``application-autoscaling``: [``botocore``] Update application-autoscaling client to latest version * api-change:``workspaces``: [``botocore``] Update workspaces client to latest version * api-change:``codecommit``: [``botocore``] Update codecommit client to latest version ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/boto3 - Changelog: https://pyup.io/changelogs/boto3/ - Repo: https://github.com/boto/boto3 </details> ### Update [Django](https://pypi.org/project/Django) from **2.0.10** to **2.0.12**. <details> <summary>Changelog</summary> ### 2.0.12 ``` =========================== *February 11, 2019* Django 2.0.12 fixes a packaging error in 2.0.11. Bugfixes ======== * Corrected packaging error from 2.0.11 (:ticket:`30175`). =========================== ``` ### 2.0.11 ``` =========================== *February 11, 2019* Django 2.0.11 fixes a security issue in 2.0.10. CVE-2019-6975: Memory exhaustion in ``django.utils.numberformat.format()`` -------------------------------------------------------------------------- If ``django.utils.numberformat.format()`` -- used by ``contrib.admin`` as well as the the ``floatformat``, ``filesizeformat``, and ``intcomma`` templates filters -- received a ``Decimal`` with a large number of digits or a large exponent, it could lead to significant memory usage due to a call to ``'{:f}'.format()``. To avoid this, decimals with more than 200 digits are now formatted using scientific notation. =========================== ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/django - Changelog: https://pyup.io/changelogs/django/ - Homepage: https://www.djangoproject.com/ </details> ### Update [pytest-testrail](https://pypi.org/project/pytest-testrail) from **2.3.2** to **2.3.3**. <details> <summary>Changelog</summary> </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pytest-testrail - Changelog: https://pyup.io/changelogs/pytest-testrail/ - Repo: http://github.com/allankp/pytest-testrail/ </details> ### Update [pip](https://pypi.org/project/pip) from **19.0.1** to **19.0.2**. <details> <summary>Changelog</summary> ### 19.0.2 ``` =================== Bug Fixes --------- - Fix a crash where PEP 517-based builds using ``--no-cache-dir`` would fail in some circumstances with an ``AssertionError`` due to not finalizing a build directory internally. (`6197 <https://github.com/pypa/pip/issues/6197>`_) - Provide a better error message if attempting an editable install of a directory with a ``pyproject.toml`` but no ``setup.py``. (`6170 <https://github.com/pypa/pip/issues/6170>`_) - The implicit default backend used for projects that provide a ``pyproject.toml`` file without explicitly specifying ``build-backend`` now behaves more like direct execution of ``setup.py``, and hence should restore compatibility with projects that were unable to be installed with ``pip`` 19.0. This raised the minimum required version of ``setuptools`` for such builds to 40.8.0. (`6163 <https://github.com/pypa/pip/issues/6163>`_) - Allow ``RECORD`` lines with more than three elements, and display a warning. (`6165 <https://github.com/pypa/pip/issues/6165>`_) - ``AdjacentTempDirectory`` fails on unwritable directory instead of locking up the uninstall command. (`6169 <https://github.com/pypa/pip/issues/6169>`_) - Make failed uninstalls roll back more reliably and better at avoiding naming conflicts. (`6194 <https://github.com/pypa/pip/issues/6194>`_) - Ensure the correct wheel file is copied when building PEP 517 distribution is built. (`6196 <https://github.com/pypa/pip/issues/6196>`_) - The Python 2 end of life warning now only shows on CPython, which is the implementation that has announced end of life plans. (`6207 <https://github.com/pypa/pip/issues/6207>`_) Improved Documentation ---------------------- - Re-write README and documentation index (`5815 <https://github.com/pypa/pip/issues/5815>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> Co-authored-by: Renovate Bot <bot@renovateapp.com> Co-authored-by: pyup-bot <github-bot@pyup.io>
### Update [pip](https://pypi.org/project/pip) from **19.0.1** to **19.0.3**. <details> <summary>Changelog</summary> ### 19.0.3 ``` =================== Bug Fixes --------- - Fix an ``IndexError`` crash when a legacy build of a wheel fails. (`6252 <https://github.com/pypa/pip/issues/6252>`_) - Fix a regression introduced in 19.0.2 where the filename in a RECORD file of an installed file would not be updated when installing a wheel. (`6266 <https://github.com/pypa/pip/issues/6266>`_) ``` ### 19.0.2 ``` =================== Bug Fixes --------- - Fix a crash where PEP 517-based builds using ``--no-cache-dir`` would fail in some circumstances with an ``AssertionError`` due to not finalizing a build directory internally. (`6197 <https://github.com/pypa/pip/issues/6197>`_) - Provide a better error message if attempting an editable install of a directory with a ``pyproject.toml`` but no ``setup.py``. (`6170 <https://github.com/pypa/pip/issues/6170>`_) - The implicit default backend used for projects that provide a ``pyproject.toml`` file without explicitly specifying ``build-backend`` now behaves more like direct execution of ``setup.py``, and hence should restore compatibility with projects that were unable to be installed with ``pip`` 19.0. This raised the minimum required version of ``setuptools`` for such builds to 40.8.0. (`6163 <https://github.com/pypa/pip/issues/6163>`_) - Allow ``RECORD`` lines with more than three elements, and display a warning. (`6165 <https://github.com/pypa/pip/issues/6165>`_) - ``AdjacentTempDirectory`` fails on unwritable directory instead of locking up the uninstall command. (`6169 <https://github.com/pypa/pip/issues/6169>`_) - Make failed uninstalls roll back more reliably and better at avoiding naming conflicts. (`6194 <https://github.com/pypa/pip/issues/6194>`_) - Ensure the correct wheel file is copied when building PEP 517 distribution is built. (`6196 <https://github.com/pypa/pip/issues/6196>`_) - The Python 2 end of life warning now only shows on CPython, which is the implementation that has announced end of life plans. (`6207 <https://github.com/pypa/pip/issues/6207>`_) Improved Documentation ---------------------- - Re-write README and documentation index (`5815 <https://github.com/pypa/pip/issues/5815>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> ### Update [wheel](https://pypi.org/project/wheel) from **0.32.3** to **0.33.1**. <details> <summary>Changelog</summary> ### 0.33.1 ``` - Fixed the ``--build-number`` option for ``wheel pack`` not being applied ``` ### 0.33.0 ``` - Added the ``--build-number`` option to the ``wheel pack`` command - Fixed bad shebangs sneaking into wheels - Fixed documentation issue with ``wheel pack`` erroneously being called ``wheel repack`` - Fixed filenames with "bad" characters (like commas) not being quoted in ``RECORD`` (PR by Paul Moore) - Sort requirements extras to ensure deterministic builds (PR by PoncinMatthieu) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/wheel - Changelog: https://pyup.io/changelogs/wheel/ - Repo: https://github.com/pypa/wheel </details> ### Update [PyYAML](https://pypi.org/project/PyYAML) from **4.2b4** to **5.1b3**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pyyaml - Homepage: http://pyyaml.org/wiki/PyYAML </details>
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
lock
bot
added
the
auto-locked
Environment
Description
In some situations after installing packages from a requirements file this error is thrown:
Expected behavior
Install works
How to Reproduce
I am not completely sure yet... I am installing from a requirements file with pre-built packages like:
sudo /home/hadoop/venv2/bin/python -m pip install -r /home/hadoop/emr.requirements.txt --find-links=/home/hadoop/packagesPre-built packages are in
/home/hadoop/packagesOutput
The text was updated successfully, but these errors were encountered: