Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip wheel -e git+https:// does a git export instead of git clone #9273

Closed
sbidoul opened this issue Dec 14, 2020 · 10 comments · Fixed by #9274
Closed

pip wheel -e git+https:// does a git export instead of git clone #9273

sbidoul opened this issue Dec 14, 2020 · 10 comments · Fixed by #9274
Labels
C: editable Editable installations C: vcs pip's interaction with version control systems like git, svn and bzr C: wheel The wheel format and 'pip wheel' command type: bug A confirmed bug or unintended behavior
Milestone

Comments

@sbidoul
Copy link
Member

sbidoul commented Dec 14, 2020

Environment

  • pip version: 20.3
  • Python version: 3.8 (probably irrelevant)
  • OS: linux (probably irrelevant)

Description

pip wheel --no-deps -e git+https://github.com/pypa/pip-test-package#egg=pip-test-package used to keep a git clone in the src directory. Now it keeps something there, that does not have a .git directory in it.

Analysis

In progress, but a quick git bissect shows that the issue originates in the same area as #9122.

@sbidoul sbidoul added C: vcs pip's interaction with version control systems like git, svn and bzr C: wheel The wheel format and 'pip wheel' command type: bug A confirmed bug or unintended behavior labels Dec 14, 2020
@sbidoul
Copy link
Member Author

sbidoul commented Dec 14, 2020

This matters because if the package being built relies on setuptools_scm's git file finder, it does not work anymore.

@uranusjr
Copy link
Member

Does this only affect wheel, or also install?

@sbidoul
Copy link
Member Author

sbidoul commented Dec 14, 2020

install works fine. The problem comes from here:

req.update_editable(self.download_dir is None)

which was changed in #8843 (the merge of wheel_download_dir and download_dir). The changed argument is obtain which drives whether pip does a vcs clone or vcs export in update_editable.

Now it does a vcs export in wheel mode too, whereas before it did that in download mode only.

@sbidoul sbidoul changed the title pip wheel -e git+https:// does not preserve a git clone in src pip wheel -e git+https:// does a git export instead of git clone Dec 14, 2020
@sbidoul
Copy link
Member Author

sbidoul commented Dec 14, 2020

So this bug has indeed the same origin as #9122, which is that _download_should_save has been replaced by a test on the presence of download_dir, which is not equivalent.

@sbidoul
Copy link
Member Author

sbidoul commented Dec 14, 2020

I suspect this clone vs export issue can also have implications on dependency resolution, for wheel and download alike. Indeed, even in download mode, if the vcs information is not present, setuptools_scm may yield incorrect version information, influencing the resolver outcome.

@sbidoul
Copy link
Member Author

sbidoul commented Dec 14, 2020

So, a radical approach could be to always vcs clone and never vcs export, which would fix this bug, but change slightly how pip download of editable requirements works, as follow.

20.2.4
src has .git dir
20.2.4
zip has .git dir
20.3.1
src has .git dir
20.3.1
zip has .git dir
proposal
src has .git dir
proposal
zip has .git dir
pip download git+https:// n/a yes n/a yes n/a yes
pip download -e git+https:// no no no no yes (this is an improvement) yes (is this change a problem?)
pip wheel git+https:// n/a n/a n/a n/a n/a n/a
pip wheel -e git+https:// yes n/a no (this bug) n/a yes (this bug is fixed) n/a

What do people think?

@sbidoul
Copy link
Member Author

sbidoul commented Dec 23, 2020

I'm assigning this to the 20.3.4 milestone it this is a regression. #9274 implements the table above. See #9338 for the root cause analysis and discussion about possible approaches.

@sbidoul sbidoul added the C: editable Editable installations label Dec 23, 2020
@pradyunsg
Copy link
Member

pradyunsg commented Dec 23, 2020

I don't think we should have the .git folder in the zip under any circumstances.

Gah, why is pip's VCS setup so complicated. :(

@RonnyPfannschmidt
Copy link
Contributor

This would be slightly less of a issue if git archives / exports had any usable Metadata whatsoever, unfortunately they don't, anything short of a full clone is typically crippled and unusable

@pradyunsg
Copy link
Member

*made an important edit to my earlier comment, because I retyped the same thing in 3 different ways and picked the worst way to combine them :)

bors bot referenced this issue in duckinator/emanate Jan 31, 2021
215: Update pip to 21.0.1 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.3.3** to **21.0.1**.



<details>
  <summary>Changelog</summary>
  
  
   ### 21.0.1
   ```
   ===================

Bug Fixes
---------

- commands: debug: Use packaging.version.parse to compare between versions. (`9461 &lt;https://github.com/pypa/pip/issues/9461&gt;`_)
- New resolver: Download and prepare a distribution only at the last possible
  moment to avoid unnecessary network access when the same version is already
  installed locally. (`9516 &lt;https://github.com/pypa/pip/issues/9516&gt;`_)

Vendored Libraries
------------------

- Upgrade packaging to 20.9
   ```
   
  
  
   ### 21.0
   ```
   =================

Deprecations and Removals
-------------------------

- Drop support for Python 2. (`6148 &lt;https://github.com/pypa/pip/issues/6148&gt;`_)
- Remove support for legacy wheel cache entries that were created with pip
  versions older than 20.0. (`7502 &lt;https://github.com/pypa/pip/issues/7502&gt;`_)
- Remove support for VCS pseudo URLs editable requirements. It was emitting
  deprecation warning since version 20.0. (`7554 &lt;https://github.com/pypa/pip/issues/7554&gt;`_)
- Modernise the codebase after Python 2. (`8802 &lt;https://github.com/pypa/pip/issues/8802&gt;`_)
- Drop support for Python 3.5. (`9189 &lt;https://github.com/pypa/pip/issues/9189&gt;`_)
- Remove the VCS export feature that was used only with editable VCS
  requirements and had correctness issues. (`9338 &lt;https://github.com/pypa/pip/issues/9338&gt;`_)

Features
--------

- Add ``--ignore-requires-python`` support to pip download. (`1884 &lt;https://github.com/pypa/pip/issues/1884&gt;`_)
- New resolver: Error message shown when a wheel contains inconsistent metadata
  is made more helpful by including both values from the file name and internal
  metadata. (`9186 &lt;https://github.com/pypa/pip/issues/9186&gt;`_)

Bug Fixes
---------

- Fix a regression that made ``pip wheel`` do a VCS export instead of a VCS clone
  for editable requirements. This broke VCS requirements that need the VCS
  information to build correctly. (`9273 &lt;https://github.com/pypa/pip/issues/9273&gt;`_)
- Fix ``pip download`` of editable VCS requirements that need VCS information
  to build correctly. (`9337 &lt;https://github.com/pypa/pip/issues/9337&gt;`_)

Vendored Libraries
------------------

- Upgrade msgpack to 1.0.2.
- Upgrade requests to 2.25.1.

Improved Documentation
----------------------

- Render the unreleased pip version change notes on the news page in docs. (`9172 &lt;https://github.com/pypa/pip/issues/9172&gt;`_)
- Fix broken email link in docs feedback banners. (`9343 &lt;https://github.com/pypa/pip/issues/9343&gt;`_)


.. note

    You should *NOT* be adding new change log entries to this file, this
    file is managed by towncrier. You *may* edit previous change logs to
    fix problems like typo corrections or such.

    To add a new change log entry, please see
        https://pip.pypa.io/en/latest/development/contributing/#news-entries

.. towncrier release notes start
   ```
   
  
  
   ### 20.3.4
   ```
   ===================

Features
--------

- ``pip wheel`` now verifies the built wheel contains valid metadata, and can be
  installed by a subsequent ``pip install``. This can be disabled with
  ``--no-verify``. (`9206 &lt;https://github.com/pypa/pip/issues/9206&gt;`_)
- Improve presentation of XMLRPC errors in pip search. (`9315 &lt;https://github.com/pypa/pip/issues/9315&gt;`_)

Bug Fixes
---------

- Fixed hanging VCS subprocess calls when the VCS outputs a large amount of data
  on stderr. Restored logging of VCS errors that was inadvertently removed in pip
  20.2. (`8876 &lt;https://github.com/pypa/pip/issues/8876&gt;`_)
- Fix error when an existing incompatibility is unable to be applied to a backtracked state. (`9180 &lt;https://github.com/pypa/pip/issues/9180&gt;`_)
- New resolver: Discard a faulty distribution, instead of quitting outright.
  This implementation is taken from 20.2.2, with a fix that always makes the
  resolver iterate through candidates from indexes lazily, to avoid downloading
  candidates we do not need. (`9203 &lt;https://github.com/pypa/pip/issues/9203&gt;`_)
- New resolver: Discard a source distribution if it fails to generate metadata,
  instead of quitting outright. This implementation is taken from 20.2.2, with a
  fix that always makes the resolver iterate through candidates from indexes
  lazily, to avoid downloading candidates we do not need. (`9246 &lt;https://github.com/pypa/pip/issues/9246&gt;`_)

Vendored Libraries
------------------

- Upgrade resolvelib to 0.5.4.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 4, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
C: editable Editable installations C: vcs pip's interaction with version control systems like git, svn and bzr C: wheel The wheel format and 'pip wheel' command type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants