Print provenance information when pip-compile fails #837

jakevdp · 2019-06-26T00:00:22Z

For motivation, see #836

For the following requirements.in file:

tensorflow-data-validation
google-cloud-bigquery~=1.14.0

Before this change, the output is this:

$ python -m piptools compile requirements.in
Could not find a version that matches google-cloud-bigquery<1.7.0,>=1.6.0,~=1.14.0
Tried: 0.20.0, 0.20.0, 0.21.0, 0.21.0, 0.22.0, 0.22.0, 0.22.1, 0.22.1, 0.23.0, 0.23.0, 0.24.0, 0.24.0, 0.25.0, 0.25.0, 0.26.0, 0.26.0, 0.27.0, 0.27.0, 0.28.0, 0.28.0, 0.29.0, 0.29.0, 0.30.0, 0.30.0, 0.31.0, 0.31.0, 0.32.0, 0.32.0, 1.0.0, 1.0.0, 1.1.0, 1.1.0, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.6.0, 1.6.0, 1.6.1, 1.6.1, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.8.1, 1.8.1, 1.9.0, 1.9.0, 1.10.0, 1.10.0, 1.11.1, 1.11.1, 1.11.2, 1.11.2, 1.11.3, 1.11.3, 1.12.0, 1.12.0, 1.12.1, 1.12.1, 1.13.0, 1.13.0, 1.14.0, 1.14.0, 1.15.0, 1.15.0
There are incompatible versions in the resolved dependencies.

With this change, the output is this:

$ python -m piptools compile requirements.in
Could not find a version that matches google-cloud-bigquery<1.7.0,>=1.6.0,~=1.14.0 (from -r requirements.in (line 2))
Tried: 0.20.0, 0.20.0, 0.21.0, 0.21.0, 0.22.0, 0.22.0, 0.22.1, 0.22.1, 0.23.0, 0.23.0, 0.24.0, 0.24.0, 0.25.0, 0.25.0, 0.26.0, 0.26.0, 0.27.0, 0.27.0, 0.28.0, 0.28.0, 0.29.0, 0.29.0, 0.30.0, 0.30.0, 0.31.0, 0.31.0, 0.32.0, 0.32.0, 1.0.0, 1.0.0, 1.1.0, 1.1.0, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.6.0, 1.6.0, 1.6.1, 1.6.1, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.8.1, 1.8.1, 1.9.0, 1.9.0, 1.10.0, 1.10.0, 1.11.1, 1.11.1, 1.11.2, 1.11.2, 1.11.3, 1.11.3, 1.12.0, 1.12.0, 1.12.1, 1.12.1, 1.13.0, 1.13.0, 1.14.0, 1.14.0, 1.15.0, 1.15.0
There are incompatible versions in the resolved dependencies:
  google-cloud-bigquery~=1.14.0 (from -r requirements.in (line 2))
  google-cloud-bigquery<1.7.0,>=1.6.0 (from apache-beam[gcp]==2.13.0->tensorflow-data-validation==0.13.1->-r requirements.in (line 1))

This involves adding a piptools-specific attribute to combined InstallRequirements objects. This might be cleaner if we used a class-based approach (defining, e.g. a CombinedInstallRequirements class) but this is difficult to do cleanly because pip's InstallRequirements class is an internal implementation that changes often between pip versions.

Changelog-friendly one-liner: Print provenance information when pip-compile fails (Fixes #836)

Contributor checklist

Provided the tests for the changes.
Requested a review from another contributor.
Gave a clear one-line description in the PR (that the maintainers can add to CHANGELOG.md on release).
Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

jakevdp · 2019-06-26T00:06:15Z

Test failures are from provenance info added to string representations of requirements. Should be straightforward to fix if the approach here seems like the right one.

codecov · 2019-06-27T17:21:02Z

Codecov Report

Merging #837 into master will decrease coverage by 0.13%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #837      +/-   ##
==========================================
- Coverage   98.69%   98.55%   -0.14%     
==========================================
  Files          36       36              
  Lines        2139     2150      +11     
  Branches      279      284       +5     
==========================================
+ Hits         2111     2119       +8     
- Misses         16       17       +1     
- Partials       12       14       +2

Impacted Files	Coverage Δ
tests/test_minimal_upgrade.py	`100% <ø> (ø)`	⬆️
tests/test_resolver.py	`100% <ø> (ø)`	⬆️
piptools/resolver.py	`100% <100%> (ø)`	⬆️
piptools/exceptions.py	`86.84% <100%> (+0.73%)`	⬆️
piptools/repositories/pypi.py	`91.05% <0%> (-1.58%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7824d94...6b0eab3. Read the comment docs.

codecov · 2019-06-27T17:21:02Z

Codecov Report

Merging #837 into master will increase coverage by 0.06%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #837      +/-   ##
==========================================
+ Coverage   98.69%   98.75%   +0.06%     
==========================================
  Files          36       36              
  Lines        2139     2173      +34     
  Branches      279      285       +6     
==========================================
+ Hits         2111     2146      +35     
  Misses         16       16              
+ Partials       12       11       -1

Impacted Files	Coverage Δ
tests/test_minimal_upgrade.py	`100% <ø> (ø)`	⬆️
piptools/resolver.py	`100% <100%> (ø)`	⬆️
piptools/exceptions.py	`89.47% <100%> (+3.36%)`	⬆️
tests/test_resolver.py	`100% <100%> (ø)`	⬆️
tests/conftest.py	`97.26% <100%> (+2.81%)`	⬆️
tests/test_sync.py	`98.74% <0%> (-1.26%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7824d94...83ceb93. Read the comment docs.

jakevdp · 2019-06-27T18:06:08Z

I'm having trouble testing the generated exception message, because the current version of NoCandidateFound used in the tests is improperly instantiated (it will need some sort of repository mock in order to be used correctly).

jakevdp · 2019-06-27T21:22:12Z

I managed to fix the issue in conftest, so that I could add an integration test of the new error behavior.

jakevdp · 2019-06-27T21:25:57Z

This is ready for a review

jakevdp · 2019-06-27T22:53:44Z

@atugushev would you be willing to review? Thanks!

atugushev

This is awesome! I've reviewed code briefly though and found a few minor nitpicks. Soon i'll review combine_install_requirements more closely and give you a comprehensive feedback.

Also i've tried PR locally with echo "pyrax==1.9.8" | pip-compile - -vo- (thanks for good example #693) and got this:

Could not find a version that matches pbr!=2.1.0,<2.0,>=1.6,>=2.0.0 (from python-novaclient==2.27.0->pyrax==1.9.8->-r /var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/tmpnu5oa32f (line 1))
Tried: 0.5.2.5.g5b3e942, 0.5.0, 0.5.1, 0.5.2, 0.5.4, 0.5.5, 0.5.6, 0.5.7, 0.5.8, 0.5.10, 0.5.11, 0.5.12, 0.5.13, 0.5.14, 0.5.15, 0.5.16, 0.5.17, 0.5.18, 0.5.19, 0.5.20, 0.5.21, 0.5.22, 0.5.23, 0.6, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.9.0, 0.10.0, 0.10.0, 0.10.1, 0.10.1, 0.10.2, 0.10.2, 0.10.3, 0.10.3, 0.10.4, 0.10.4, 0.10.5, 0.10.5, 0.10.6, 0.10.6, 0.10.7, 0.10.7, 0.10.8, 0.10.8, 0.11.0, 0.11.0, 0.11.1, 0.11.1, 1.0.0, 1.0.0, 1.0.1, 1.0.1, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.8.1, 1.8.1, 1.9.0, 1.9.0, 1.9.1, 1.9.1, 1.10.0, 1.10.0, 2.0.0, 2.0.0, 2.1.0, 2.1.0, 3.0.0, 3.0.0, 3.0.1, 3.0.1, 3.1.0, 3.1.0, 3.1.1, 3.1.1, 4.0.0, 4.0.0, 4.0.1, 4.0.1, 4.0.2, 4.0.2, 4.0.3, 4.0.3, 4.0.4, 4.0.4, 4.1.0, 4.1.0, 4.1.1, 4.1.1, 4.2.0, 4.2.0, 4.3.0, 4.3.0, 5.0.0, 5.0.0, 5.1.0, 5.1.0, 5.1.1, 5.1.1, 5.1.2, 5.1.2, 5.1.3, 5.1.3, 5.2.0, 5.2.0, 5.2.1, 5.2.1, 5.3.0, 5.3.0, 5.3.1, 5.3.1
There are incompatible versions in the resolved dependencies.
- pbr!=2.1.0,>=2.0.0 (from oslo.utils==3.41.0->python-novaclient==2.27.0->pyrax==1.9.8->-r /var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/tmpnu5oa32f (line 1))
- pbr!=2.1.0,>=2.0.0 (from oslo.serialization==2.29.1->python-novaclient==2.27.0->pyrax==1.9.8->-r /var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/tmpnu5oa32f (line 1))
- pbr!=2.1.0,>=2.0.0 (from oslo.i18n==3.23.1->python-novaclient==2.27.0->pyrax==1.9.8->-r /var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/tmpnu5oa32f (line 1))
- pbr!=2.1.0,>=2.0.0 (from python-keystoneclient==3.19.0->python-novaclient==2.27.0->pyrax==1.9.8->-r /var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/tmpnu5oa32f (line 1))
- pbr<2.0,>=1.6 (from python-novaclient==2.27.0->pyrax==1.9.8->-r /var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/tmpnu5oa32f (line 1))

Looks tasty!

piptools/exceptions.py

tests/test_resolver.py

piptools/exceptions.py

piptools/resolver.py

georgek · 2019-07-01T09:37:05Z

This looks good. I began to work on this myself and also thought using a class like CombinedInstallRequirements would be a good approach (ie. one that looks like a normal InstallRequirements to pip, but has extra info in for pip-tools' benefit). However, I did not take into account that it was a pip internal that is subject to change so this might be a better approach (for now).

atugushev

LGTM! 👍

atugushev · 2019-07-01T09:51:11Z

@jakevdp could you please squash your commits before it'll be merged?

jazzband#836)

jakevdp · 2019-07-01T15:15:08Z

squashed & force-pushed

jakevdp · 2019-07-01T18:14:30Z

Awesome, thanks everyone!

atugushev · 2019-07-17T22:31:08Z

pip-tools v3.9.0 is released.

andersk · 2019-08-25T01:31:43Z

According to git bisect, this was responsible for breaking #851. Can you take a look?

andersk · 2019-08-25T02:03:30Z

Just based on taking a look at what changed here, deleting source_ireqs.sort(key=str) fixes the test case in #851. Is that the right solution?

Not sure if this is the right solution, but this seems to fix jazzband#851, which was introduced when the `source_ireqs.sort(key=str)` operation was added in jazzband#837. Signed-off-by: Anders Kaseorg <andersk@mit.edu>

vphilippon · 2019-10-01T14:48:56Z

I just stumbled on this new feature (I've been away for a while), you are my heroes! 🎉 🎉 🎉

jakevdp changed the title ~~Print provenance information when pip-compile fails~~ WIP: Print provenance information when pip-compile fails Jun 26, 2019

jakevdp mentioned this pull request Jun 26, 2019

Showing provenance of conflicting constraints in pip-compile #836

Closed

jakevdp changed the title ~~WIP: Print provenance information when pip-compile fails~~ Print provenance information when pip-compile fails Jun 27, 2019

atugushev added this to the 3.9.0 milestone Jun 27, 2019

atugushev added the enhancement Improvements to functionality label Jun 27, 2019

atugushev reviewed Jun 27, 2019

View reviewed changes

piptools/exceptions.py Outdated Show resolved Hide resolved

piptools/exceptions.py Outdated Show resolved Hide resolved

tests/test_resolver.py Outdated Show resolved Hide resolved

piptools/exceptions.py Outdated Show resolved Hide resolved

auvipy approved these changes Jun 29, 2019

View reviewed changes

atugushev reviewed Jun 29, 2019

View reviewed changes

piptools/resolver.py Outdated Show resolved Hide resolved

piptools/resolver.py Show resolved Hide resolved

atugushev approved these changes Jul 1, 2019

View reviewed changes

ENH: print package provenance information when pip-compile fails (fixes

83ceb93

jazzband#836)

jakevdp force-pushed the provenance branch from e1e453a to 83ceb93 Compare July 1, 2019 15:14

atugushev merged commit b37b91d into jazzband:master Jul 1, 2019

jakevdp deleted the provenance branch July 1, 2019 17:51

georgek mentioned this pull request Jul 6, 2019

Show where a conflict is coming from #473

Closed

kdmccormick mentioned this pull request Jul 17, 2019

Release 3.9.0 #850

Closed

andersk mentioned this pull request Aug 25, 2019

3.9.0 pip-compile gets confused by GitHub installs without -e #851

Closed

andersk mentioned this pull request Aug 25, 2019

Resolver.constraints: Keep our_constraints before their_constraints #879

Merged

4 tasks

atugushev mentioned this pull request Sep 26, 2019

future: display more information / source with NoCandidateFound error #165

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Print provenance information when pip-compile fails #837

Print provenance information when pip-compile fails #837

jakevdp commented Jun 26, 2019 •

edited

Loading

jakevdp commented Jun 26, 2019

codecov bot commented Jun 27, 2019

codecov bot commented Jun 27, 2019 •

edited

Loading

jakevdp commented Jun 27, 2019

jakevdp commented Jun 27, 2019

jakevdp commented Jun 27, 2019

jakevdp commented Jun 27, 2019

atugushev left a comment

georgek commented Jul 1, 2019

atugushev left a comment

atugushev commented Jul 1, 2019

jakevdp commented Jul 1, 2019

jakevdp commented Jul 1, 2019

atugushev commented Jul 17, 2019

andersk commented Aug 25, 2019

andersk commented Aug 25, 2019

vphilippon commented Oct 1, 2019

Print provenance information when pip-compile fails #837

Print provenance information when pip-compile fails #837

Conversation

jakevdp commented Jun 26, 2019 • edited Loading

Contributor checklist

jakevdp commented Jun 26, 2019

codecov bot commented Jun 27, 2019

Codecov Report

codecov bot commented Jun 27, 2019 • edited Loading

Codecov Report

jakevdp commented Jun 27, 2019

jakevdp commented Jun 27, 2019

jakevdp commented Jun 27, 2019

jakevdp commented Jun 27, 2019

atugushev left a comment

Choose a reason for hiding this comment

georgek commented Jul 1, 2019

atugushev left a comment

Choose a reason for hiding this comment

atugushev commented Jul 1, 2019

jakevdp commented Jul 1, 2019

jakevdp commented Jul 1, 2019

atugushev commented Jul 17, 2019

andersk commented Aug 25, 2019

andersk commented Aug 25, 2019

vphilippon commented Oct 1, 2019

jakevdp commented Jun 26, 2019 •

edited

Loading

codecov bot commented Jun 27, 2019 •

edited

Loading