Fix dist_utils.py so it doesn't rely on internal pip API

[Kami]

This pull request fixes dist_utils.py module so it doesn't rely on internal pip functionality and so it works with latest pip version.

Background, Context

While working on another project which has code similar to our dist_utils.fetch_requirements I noticed it fails with newer versions of pip with the following error:

    Failed to import parse_requirements from pip: cannot import name pep425tags
    Using pip: 19.2.1

In our case, the error itself could manifest itself when installing st2client using a specific version of pip.

Main problem is that the code in dist_utils.py depends on internal pip functionality which can change across the pip versions (and it already changed in the past and that caused us quite some trouble back then).

Proposed Solution

This solution re-implements fetch_requirements so it doesn't depend on the pip anymore.

Internal pip parse_requirements does a lot of more stuff, but we actually only care about a simple use case - we only want a list of simple and URL requirements back and that's trivial to implement ourselves.

[arm4b]

I'm wondering how error-prone is that code block?
Did you take the rules from pip upstream or it's something else you consulted for code logic?

[Kami]

I didn't check pip code itself, but I compared the results and consulted high level documentation on supported requirements format (which is the only thing I care about at this point - the function output).

If you check the tests, you will see we have tests which compare the result of our version and pip version to ensure they are the same.

It's likely there are still some edge cases somewhere, but it's not a big deal at this point since for our current requirements.txt, the end result is the same.

[blag]

This code did not support environment markers from pip, so I had to add (hacky) support for them in #4895.

@Kami Why can the dist_utils.py module not use any third party dependencies?

# NOTE: This script can't rely on any 3rd party dependency so we need to use this code here

Instead of importing from pip directly (which we should never have been doing in the first place, IMO), if we can use third party dependencies, we can utilize the packaging package. It has an entire section devoted to reading and parsing requirements.txt files, and that would be kept up-to-date with what pip and setuptools support, and would lower our maintenance burden of this code.

[arm4b]

👍

Would be great to port this logic to https://github.com/StackStorm/st2-packages/blob/master/packages/st2/dist_utils.py as well

[punkrokk]

Thank you! This has been a thing in parts of my world at times.

[punkrokk]

@blag is the v3.1.1 tag correct?

[blag]

@punkrokk I originally created the v3.1.1 to track PRs that were good candidates for a v3.1.1 bugfix release, but @m4dcoder and @armab disagreed on their usefulness, so I deleted the tag itself. That removed it from the list of tags on this PR, but apparently just deleting the tag doesn't create a "GitHub PR changelog" entry for removing the tag from this PR.

The reason I created a tag for bugfix versions is that you can only assign PRs to one milestone. Which kinda sorta makes some sense, but I do think we should have some way of tracking PRs that would be good candidates for the next bugfix release/s. At this point, however, that's NMFP.

[nmaludy]

This just bit us.

Need to copy this to other places where we copy/paste dist_utils.py such as: https://github.com/StackStorm/st2-auth-backend-flat-file