Showing provenance of conflicting constraints in pip-compile

[jakevdp]

Thanks for the great tool. I use pip-compile to manage a large production Python environment, and one of the main pain-points I've found is that when an upstream change leads to a conflict, it can often be quite difficult to ascertain the cause of the failure.

For example, consider the following requirements.in file, which in my case can have several dozen listed dependencies:

# requirements.in
# ... several dozen packages
google-cloud-bigquery~=1.14.0
# ... several dozen more packages
tensorflow-data-validation~=0.13.1
# ... several dozen more packages

If you run this through pip compile, it finds a conflict:

$ python -m piptools compile requirements.in
Could not find a version that matches google-cloud-bigquery<1.7.0,>=1.6.0,~=1.14.0
Tried: 0.20.0, 0.20.0, 0.21.0, 0.21.0, 0.22.0, 0.22.0, 0.22.1, 0.22.1, 0.23.0, 0.23.0, 0.24.0, 0.24.0, 0.25.0, 0.25.0, 0.26.0, 0.26.0, 0.27.0, 0.27.0, 0.28.0, 0.28.0, 0.29.0, 0.29.0, 0.30.0, 0.30.0, 0.31.0, 0.31.0, 0.32.0, 0.32.0, 1.0.0, 1.0.0, 1.1.0, 1.1.0, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.6.0, 1.6.0, 1.6.1, 1.6.1, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.8.1, 1.8.1, 1.9.0, 1.9.0, 1.10.0, 1.10.0, 1.11.1, 1.11.1, 1.11.2, 1.11.2, 1.11.3, 1.11.3, 1.12.0, 1.12.0, 1.12.1, 1.12.1, 1.13.0, 1.13.0, 1.14.0, 1.14.0, 1.15.0, 1.15.0
There are incompatible versions in the resolved dependencies.

but this error contains no information about the provenance of the conflicting constraints, and so it is not much help in ascertaining the cause of the conflict.

I have a partial solution to this in jakevdp@c33cafc; it works by propagating combined comes_from metadata as a string in the combined constraints. The result for the above requirements looks like this:

$ python -m piptools compile requirements.in
Could not find a version that matches google-cloud-bigquery<1.7.0,>=1.6.0,~=1.14.0 (from apache-beam[gcp]==2.13.0->tensorflow-data-validation==0.13.1->-r requirements.in (line 5), -r requirements.in (line 3))
Tried: 0.20.0, 0.20.0, 0.21.0, 0.21.0, 0.22.0, 0.22.0, 0.22.1, 0.22.1, 0.23.0, 0.23.0, 0.24.0, 0.24.0, 0.25.0, 0.25.0, 0.26.0, 0.26.0, 0.27.0, 0.27.0, 0.28.0, 0.28.0, 0.29.0, 0.29.0, 0.30.0, 0.30.0, 0.31.0, 0.31.0, 0.32.0, 0.32.0, 1.0.0, 1.0.0, 1.1.0, 1.1.0, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.6.0, 1.6.0, 1.6.1, 1.6.1, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.8.1, 1.8.1, 1.9.0, 1.9.0, 1.10.0, 1.10.0, 1.11.1, 1.11.1, 1.11.2, 1.11.2, 1.11.3, 1.11.3, 1.12.0, 1.12.0, 1.12.1, 1.12.1, 1.13.0, 1.13.0, 1.14.0, 1.14.0, 1.15.0, 1.15.0
There are incompatible versions in the resolved dependencies.

It's not perfect or pretty, but it makes debugging these kinds of dependency conflicts much easier.

My solution is admittedly a bit of a hack (which is why this is an issue rather than a pull request) but I was wondering if there are better solutions I am overlooking, or if you'd be open to a contribution along these lines if I put in some effort to clean up my approach.

Thanks!

[atugushev]

Hello @jakevdp,

Thanks for raising this issue! I would love to see this feature in pip-tools. Please, don't hesitate to shoot a "WIP" PR. I hope we can implement it cooperatively.

[jakevdp]

Put together a prototype in #837, with some improvements over my initial approach.