Fix issue #1433: parse requirements in markers #1472

vstinner · 2014-01-14T17:39:56Z

InstallRequirement supports PEP 426 markers
RequirementSet.add_requirement() ignores an InstallRequirement if
markers don't match.

vstinner · 2014-01-14T17:40:54Z

Example of requirements for my Trollius project:

futures; python_version < '2.7'
mock; python_version < '3.3'
nose
ordereddict; python_version < '2.7'
unittest2; python_version < '2.7'

Output with Python 2.7:

$ python2.7 -c 'import pip; pip.main()' install -r req.txt 
Ignore futures: markers "python_version < '2.7'" don't match
Ignore ordereddict: markers "python_version < '2.7'" don't match
Ignore unittest2: markers "python_version < '2.7'" don't match
Requirement already satisfied (use --upgrade to upgrade): mock in /usr/lib/python2.7/site-packages (from -r req.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): nose in /usr/lib/python2.7/site-packages (from -r req.txt (line 3))
Cleaning up...

Trollius: https://bitbucket.org/haypo/trollius/

dholth · 2014-01-14T18:14:14Z

The original markers specification explicitly did not support < and > against strings. Does it now?

qwcode · 2014-01-14T18:27:52Z

in PEP426 it says:

MARKER: EXPR [(and|or) EXPR]*
EXPR: ("(" MARKER ")") | (SUBEXPR [CMPOP SUBEXPR])
CMPOP: (==|!=|<|>|<=|>=|in|not in)

I'm inclined to think we need "Requirement Files 2.0" for this. json format.
it would also help with #790

vstinner · 2014-01-15T16:05:55Z

The new commit should fix tests on Travis. I'm not sure that "; " is the best separator for markers. Do you have a better proposition? The semicolon is not a random choice, it comes from setup.cfg.

vstinner · 2014-01-23T09:03:57Z

@qwcode: Sorry, but I don't understand what do you mean by:

I'm inclined to think we need "Requirement Files 2.0" for this. json format.

qwcode · 2014-01-23T17:10:40Z

@Haypo see the thread I started last night on pypa-dev about this
https://groups.google.com/forum/#!topic/pypa-dev/0zI8Jw-PYVk

vstinner · 2014-02-12T16:56:11Z

I just rebased the patch on lastest develop branch.

dholth · 2014-02-12T19:50:09Z

+1. Unlike Requirements 2.0 this is simple and it exists.

vstinner · 2014-03-03T15:02:56Z

With my latest change, the syntax is now:

mock >= 0.9 # comment
{'name': 'mock >= 0.9'} # comment works here too
{'name': 'mock3', 'markers': 'python_version >= "3"'} # markers!
# {...} syntax opens the door for more options like "install-options" and "global-options" as well

msabramo · 2014-05-14T17:23:57Z

👍

This would be very handy to be able to make requirements conditional. I've often wished for this when porting packages to Python 3.

Ivoz · 2014-05-14T20:03:22Z

I would prefer to leave out anything to do with {} and code eval until a requirements 2.0. i.e anything other than the first code example in #1472 (comment)

dholth · 2014-05-22T13:31:10Z

This feature should be merged. I agree with just allowing the ; for now, until we have time to overengineer a requirements 2.0.

vstinner · 2014-05-22T15:35:41Z

Ok, here is a new proposal of syntax for markers in requirements. Use the semicolon as separator for the common code, BUT requires semicolon+space separator if the line starts with an URL.

Valid syntax:

# semicolon + space separator
futures; python_version < '2.7'
# semicolon without space works too
mock;python_version < '3.3'
# URL with semicolon + space separator
http://foo.com/?p=bar.git;a=snapshot;h=v0.1;sf=tgz; python_version < '3.3'

Invalid syntax, ;python_version < '3.3' is part of the URL:

# URL with semicolon separator
http://foo.com/?p=bar.git;a=snapshot;h=v0.1;sf=tgz;python_version < '3.3'

These examples with URL are ugly but it was just to show you the worst case, when the URL also contains semicolons. A more common case looks like this:

six
futures; python_version < '2.7'
https://example.com/foo.tar.gz
https://example.com/bar.tar.gz; python_version < '2.7'

msabramo · 2014-05-22T15:59:09Z

👍

dholth · 2014-05-22T16:15:38Z

I've done something like this, except the rule was "the last semicolon
in the line of text is the separator".

On Thu, May 22, 2014, at 11:59 AM, Marc Abramowitz wrote:

👍

—

Reply to this email directly or [1]view it on GitHub.
[208018__eyJzY29wZSI6Ik5ld3NpZXM6QmVhY29uIiwiZXhwaXJlcyI6MTcxNjM5MzU0OS
wiZGF0YSI6eyJpZCI6MjM1MTEyNjV9fQ==--b3d853556e0ecc5f5444ee106bd2305f15d
54e3a.gif]

References

Fix issue #1433: parse requirements in markers #1472 (comment)

vstinner · 2014-05-22T19:27:19Z

I've done something like this, except the rule was "the last semicolon in the line of text is the separator".

Oh, it makes me realize that it would be nice to add a test checking that a marker can contain a semicolon. Travis was unhappy. There was a obvious bug in the method checking if markers match or not. It's now fixed but I also added unit tests on this method.

The whole change now has a nice coverage in term of unit tests, and I'm happy with the new syntax. So it's ready for a final review ;-)

vstinner · 2014-05-24T00:34:24Z

" Error — The Travis CI build could not complete due to an error · Details "

It looks like Python 3 tests are too slow for Travis. I played the tests manually on my PC and py33 tests pass: "435 passed, 8 skipped in 803.33 seconds".

vstinner · 2014-10-14T12:50:22Z

Hey, did I forgot something to get my patch merged? What should I do?

matrixise · 2014-10-14T14:57:45Z

+1 for this feature.

hobbestigrou · 2014-10-14T15:16:08Z

I agree also for this feature.

mlhamel · 2014-10-14T15:42:21Z

+1, that would be awesome and would help limiting the code in, let's say... setup.py by example

pfmoore · 2014-10-14T16:28:06Z

@Haypo looks like the patch needs rebasing. Also, it could do with documentation.

@mihamel how would it affect setup.py? This specifically doesn't apply to setup_requires, so won't be useful for project dependencies. You'll need PEP 426 support in pip and setuptools for that.

mlhamel · 2014-10-14T17:53:45Z

@pfmoore it doesn't, i was just pointing the fact that people are using setup.py for doing that kind of things, since it's python code. but sometime, you end up having a huge file which is super difficult to maintain. Being able to do it in a simpler way in a requirement.txt would be fantastic.

Take this one by example:

https://github.com/odoo/odoo/blob/6.0/setup.py

(it's way more better in their latest version although)

dstufft · 2014-10-14T17:56:14Z

setup.py and requirements.txt are for different things.

https://caremad.io/blog/setup-vs-requirement/

mlhamel · 2014-10-14T17:57:47Z

@dstufft yeah but people are using both. for different reason, that would be another reason to use requirement.txt for me... :)

* InstallRequirement supports PEP 426 markers * RequirementSet.add_requirement() ignores an InstallRequirement if markers don't match.

dholth · 2014-10-14T19:47:04Z

It is possible to use markers in setup.py by using a colon separator in the extra name. Wheel's own setup.py demonstrates. The empty extra name ':python_version=="2.6"' is how you do markers on dependencies that would otherwise go into install_requires = [].

  extras_require={
      ':python_version=="2.6"': ['argparse'],
      'signatures': ['keyring'],
      'signatures:sys_platform!="win32"': ['pyxdg'],
      'faster-signatures': ['ed25519ll'],
      'tool': []
      },

pfmoore · 2014-10-14T20:01:16Z

I'd rather we get a properly documented and supported solution. @dholth I've no idea if this is supported (I've no doubt it works). Is it documented anywhere?

In retrospect, and given the confusion the setup.py vs requirements question has caused here, I'd rather see a properly agreed spec for using markers in "requirements" in the general sense, and an implementation that adds them to both setuptools (so they'll work in setup.py) and pip (so they'll be parsed from both install_requires and requirements.txt.

vstinner · 2014-10-14T20:33:16Z

@Haypo looks like the patch needs rebasing.

Done.

Also, it could do with documentation.

I added something to the documentation, is it enough?

I ran "tox -e pep8,docs,py27,py33" with success. Let's see if Travis agrees with me ;-)

berkerpeksag · 2014-10-15T07:57:09Z

docs/reference/pip_install.rst

@@ -29,6 +29,12 @@ and like arguments to :ref:`pip install`, the following forms are supported::
    [-e] <local project path>
    [-e] <vcs project url>

+Since the version 6.0, pip also supports markers using the "; " separator.


I guess "version 6.0" means "setuptools 6.0", right? :)

I guess "version 6.0" means "setuptools 6.0", right? :)

I searched for the version of pip and I found version = "6.0.dev1" in pip/init.py. It is not the version of pip?

Correct - we removed the initial "1." so version 6.X is what was previously being referred to as 1.6.X. By the way, a minor nit, "Since version 6.0" would be better grammatically.

matrixise · 2014-10-15T08:00:49Z

Please, merge it. Thank you

matrixise · 2014-10-15T08:02:54Z

Example, OrderedDict is in Python 2.7 and Python 3.x but not in Python 2.6
On some Redhat or CentOS servers, it's Python 2.6.

It's now possible to specify requirements markers in requirements. Examples:: futures; python_version < '2.7' mock; python_version < '3.3' nose ordereddict; python_version < '2.7' unittest2; python_version < '2.7' The separator is "; ". For convinience, ";" alone is also supported, but no in URLs. The ";" character is a legit and common character in an URL. Example of valid URL without markers:: http://foo.com/?p=bar.git;a=snapshot;h=v0.1;sf=tgz Example of URL with markers:: http://foo.com/?p=bar.git;a=snapshot;h=v0.1;sf=tgz; python_version < '3.3'

vstinner · 2014-10-29T14:39:20Z

I ran "tox -e pep8,docs,py27,py33" with success. Let's see if Travis agrees with me ;-)

Travis agrees with me. So what's the next step?

matrixise · 2014-11-20T17:42:13Z

Thanks

vstinner mentioned this pull request Jan 28, 2014

Running unit tests with tox writes more than 5 GB in /tmp #1512

Closed

qwcode mentioned this pull request Mar 5, 2014

support version of python in the requirements file #767

Closed

Fix issue #1433: parse requirements in markers

451c8ee

* InstallRequirement supports PEP 426 markers * RequirementSet.add_requirement() ignores an InstallRequirement if markers don't match.

berkerpeksag reviewed Oct 15, 2014
View reviewed changes

dstufft added this to the 6.0 milestone Nov 20, 2014

dstufft mentioned this pull request Nov 20, 2014

Haypo req markers #2134

Merged

dstufft closed this in #2134 Nov 20, 2014

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 4, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 4, 2019

Fix issue #1433: parse requirements in markers #1472

Fix issue #1433: parse requirements in markers #1472

Conversation

vstinner commented Jan 14, 2014

vstinner commented Jan 14, 2014

dholth commented Jan 14, 2014

qwcode commented Jan 14, 2014

vstinner commented Jan 15, 2014

vstinner commented Jan 23, 2014

qwcode commented Jan 23, 2014

vstinner commented Feb 12, 2014

dholth commented Feb 12, 2014

vstinner commented Mar 3, 2014

msabramo commented May 14, 2014

Ivoz commented May 14, 2014

dholth commented May 22, 2014

vstinner commented May 22, 2014

msabramo commented May 22, 2014

dholth commented May 22, 2014

vstinner commented May 22, 2014

vstinner commented May 24, 2014

vstinner commented Oct 14, 2014

matrixise commented Oct 14, 2014

hobbestigrou commented Oct 14, 2014

mlhamel commented Oct 14, 2014

pfmoore commented Oct 14, 2014

mlhamel commented Oct 14, 2014

dstufft commented Oct 14, 2014

mlhamel commented Oct 14, 2014

dholth commented Oct 14, 2014

pfmoore commented Oct 14, 2014

vstinner commented Oct 14, 2014

berkerpeksag Oct 15, 2014

Choose a reason for hiding this comment

vstinner Oct 15, 2014

Choose a reason for hiding this comment

pfmoore Oct 15, 2014

Choose a reason for hiding this comment

matrixise commented Oct 15, 2014

matrixise commented Oct 15, 2014

vstinner commented Oct 29, 2014

matrixise commented Nov 20, 2014