pip-compile must not archive the entire directory of a locally available editable requirement

[costypetrisor]

I'm using pip-compile for a project whose working dir is really large, and that project is passed as an editable requirement to pip-compile.
Unfortunately the code behind pip-compile which is imported from pip will make an archive of this entire large working dir even if this editable requirement is available locally.
The archive will be done in %USERPROFILE%\AppData\Local\pip-tools\Cache\pkgs or ~/.cache/pip-tools/Cache/pkgs. I found out about this when this folder bloated to almost 10 gigs.
And another ugly side effect is that I'm experiencing really slow pip-compile times (that large project's dir contains many files, so it's about 5 minutes for me).

I've fixed this by copying some code from pip's RequirementSet._prepare_file inside Resolver._iter_dependencies to avoid this call from pip just for local editable reqs.

I know this fix's approach is not that easily maintainable in the future, and because of that I'm not expecting to have this merged in pip-tools, but I just wanted to share this PR anyway, just in case other people wish to use the fix just for them, like I do.

Contributor checklist

• Provided the tests for the changes
• Added the changes to CHANGELOG.md
• Requested (or received) a review from another contributor

CHANGELOG entry: Fixed long compile durations with an locally available editable requirement

[vphilippon]

Hey @costypetrisor, thanks for putting your time into this.

I've looked at the pip call your trying to avoid, and I think that if you could simply have download_dir=None in those case, it would fix it.
So, the condition you've put to detect that we have an "editable requirement locally available" could be put in pypi.py, with the condition for VSC sources.

If you can give it a try, we could get this to be a maintainable patch, and merged.

[costypetrisor]

Hi @vphilippon

Thanks for looking at this :)
I've been blindsided when I first looked at pip's code and didn't notice that the call I was trying to avoid was under a nice if controllable from the outside.

I've updated the PR.

Thanks :)

[vphilippon]

@costypetrisor Could you rebase this on master? _compat/contextlib2 was removed.

[costypetrisor]

@vphilippon rebase done :)

[costypetrisor]

hold on a second, I think I pushed the wrong thing

[costypetrisor]

@vphilippon the correct PR branch has been pushed now :)
Thanks! :)

[vphilippon]

LGTM 👍

Also, I did a test with a larger-than-usual project myself (the tar.gz is ~180KB), and I'm seeing a 1 minute timesave on the first pip-compile with cleared cache, and no performance loss on the next pip-compile. And that's on a SSD.
Good job!