environment markers 'duplicated' (requirements + install_requires + pyproject.toml)

[altendky]

Note how the requirements markers were duplicated below. The case differs in this output but not in the source. The source varies by the requirements.in having the version specified and the install_requires not having it.

twisted==19.7.0; platform_system != "darwin" and platform_system != "windows" ; platform_system != "Darwin" and platform_system != "Windows"

Environment Versions

1. Example finalized in Linux, issue also observed in Windows and macos
2. Python version: 3.7.4
3. pip version: 19.2.2
4. pip-tools version: 4.0.0

Steps to replicate

altendky@lt:~/tmp$ for file in pyproject.toml requirements.in setup.py; do echo ---- $file; echo; cat $file; echo; done; rm -rf venv; python3 -m venv venv; venv/bin/python --version --version; venv/bin/pip install --upgrade pip wheel setuptools pip-tools; rm requirements.txt; venv/bin/pip-compile --output-file requirements.txt requirements.in

---- pyproject.toml

[build-system]
requires = [
    "setuptools",
    "wheel",
]

---- requirements.in

-e file:.#egg=xyz

Twisted==19.7.0; platform_system != "Darwin" and platform_system != "Windows"

---- setup.py

import setuptools


setuptools.setup(
    name="xyz",
    install_requires=[
        'Twisted; platform_system != "Darwin" and platform_system != "Windows"',
    ],
)

Expected result

Environment markers not duplicated. I don't know whether upper or lower case is proper.

twisted==19.7.0; platform_system != "Darwin" and platform_system != "Windows"

Actual result

Environment markers are duplicated.

Python 3.7.4 (default, Aug  2 2019, 12:58:45) 
[GCC 6.3.0 20170516]
Collecting pip
  Using cached https://files.pythonhosted.org/packages/8d/07/f7d7ced2f97ca3098c16565efbe6b15fafcba53e8d9bdb431e09140514b0/pip-19.2.2-py2.py3-none-any.whl
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/bb/10/44230dd6bf3563b8f227dbf344c908d412ad2ff48066476672f3a72e174e/wheel-0.33.4-py2.py3-none-any.whl
Collecting setuptools
  Using cached https://files.pythonhosted.org/packages/75/b3/0a106dfaf7f48aef638da80b32608617cc8de4b24a22c8cd3759c32e5d30/setuptools-41.1.0-py2.py3-none-any.whl
Collecting pip-tools
  Using cached https://files.pythonhosted.org/packages/49/91/69286bbcf185be26a1d3fa555bfa8b54ea4cb916b50485e9dfc0dd9fa638/pip_tools-4.0.0-py2.py3-none-any.whl
Collecting six (from pip-tools)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting click>=6 (from pip-tools)
  Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl
Installing collected packages: pip, wheel, setuptools, six, click, pip-tools
  Found existing installation: pip 19.0.3
    Uninstalling pip-19.0.3:
      Successfully uninstalled pip-19.0.3
  Found existing installation: setuptools 40.8.0
    Uninstalling setuptools-40.8.0:
      Successfully uninstalled setuptools-40.8.0
Successfully installed click-7.0 pip-19.2.2 pip-tools-4.0.0 setuptools-41.1.0 six-1.12.0 wheel-0.33.4
rm: cannot remove 'requirements.txt': No such file or directory
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file=requirements.txt requirements.in
#
-e file:.#egg=xyz
attrs==19.1.0             # via automat, twisted
automat==0.7.0            # via twisted
constantly==15.1.0        # via twisted
hyperlink==19.0.0         # via twisted
idna==2.8                 # via hyperlink
incremental==17.5.0       # via twisted
pyhamcrest==1.9.0         # via twisted
six==1.12.0               # via automat, pyhamcrest
twisted==19.7.0; platform_system != "darwin" and platform_system != "windows" ; platform_system != "Darwin" and platform_system != "Windows"
zope.interface==4.6.0     # via twisted

# The following packages are considered to be unsafe in a requirements file:
# setuptools==41.1.0        # via pyhamcrest, zope.interface

[atugushev]

Hello @altendky,

Thanks for submitting this! It seems you've found also the way to reproduce the #660.

[altendky]

Removing the .lower() here avoids the lowercase 'windows' and 'darwin' but doesn't avoid the duplication.

pip-tools/piptools/utils.py

Line 84 in 52b9d3e

line = str(ireq.req).lower()

Do note that the pyproject.toml is actually a part of recreating this... :|

pip-tools/piptools/utils.py

Lines 74 to 93 in 52b9d3e

	def format_requirement(ireq, marker=None, hashes=None):
	"""
	Generic formatter for pretty printing InstallRequirements to the terminal
	in a less verbose way than using its `__str__` method.
	"""
	if ireq.editable:
	line = "-e {}".format(ireq.link.url)
	elif is_url_requirement(ireq):
	line = ireq.link.url
	else:
	line = str(ireq.req).lower()
	if marker:
	line = "{} ; {}".format(line, marker)
	if hashes:
	for hash_ in sorted(hashes):
	line += " \\\n --hash={}".format(hash_)
	return line

str(ireq.req).lower() gives 'twisted==19.7.0; platform_system != "darwin" and platform_system != "windows"' but markers (from below) is also set to 'platform_system != "Darwin" and platform_system != "Windows"' thus them getting concatenated and resulting in the effective duplication.

pip-tools/piptools/writer.py

Line 168 in 52b9d3e

markers.get(key_from_ireq(ireq)),

So, easy enough to throw in a hack to 'solve' my case but... being my first foray into pip-tools code, I don't know 1) how multiple markers from different places are supposed to be handled or 2) which place the resulting net environment markers should be stored.

If someone has some guidance to help ramp me up on these points that would be appreciated. In the meantime I'll try to see how far I can get on my own (or maybe I'll take a walk and eat lunch :]).

[altendky]

One copy of the environment marker is coming from the requirements.in and the other from the setup.py. I don't know that there's a general solution to merging multiple conditions in a sensible way. Maybe there should be an error raised? I'm thinking at this point I shift back to reconsidering how I'm using pip-tools since I'm not sure that the setup I've got even makes sense. But still, something should happen other than generating an invalid requirements.txt.

[atugushev]

FYI, related to #518.

[atugushev]

This was fixed in #1567

Doesn't reproduce anymore

root@py311:/tmp/lol# for file in pyproject.toml requirements.in setup.py; do echo ---- $file; echo; cat $file; echo; done; rm -rf venv; python3 -m venv venv; venv/bin/python --version --version; venv/bin/pip install --upgrade pip wheel setuptools pip-too

---- pyproject.toml

[build-system]
requires = [
    "setuptools",
    "wheel",
]

---- requirements.in

-e file:.#egg=xyz

Twisted==19.7.0; platform_system != "Darwin" and platform_system != "Windows"

---- setup.py

import setuptools


setuptools.setup(
    name="xyz",
    install_requires=[
        'Twisted; platform_system != "Darwin" and platform_system != "Windows"',
    ],
)

Python 3.11.1 (main, Dec  8 2022, 00:05:00) [GCC 10.2.1 20210110]
Requirement already satisfied: pip in ./venv/lib/python3.11/site-packages (22.3.1)
Collecting wheel
  Using cached wheel-0.38.4-py3-none-any.whl (36 kB)
Requirement already satisfied: setuptools in ./venv/lib/python3.11/site-packages (65.5.0)
Collecting setuptools
  Using cached setuptools-65.6.3-py3-none-any.whl (1.2 MB)
Collecting pip-tools
  Using cached pip_tools-6.11.0-py3-none-any.whl (52 kB)
Collecting build
  Using cached build-0.9.0-py3-none-any.whl (17 kB)
Collecting click>=8
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting packaging>=19.0
  Using cached packaging-22.0-py3-none-any.whl (42 kB)
Collecting pep517>=0.9.1
  Using cached pep517-0.13.0-py3-none-any.whl (18 kB)
Installing collected packages: wheel, setuptools, pep517, packaging, click, build, pip-tools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 65.5.0
    Uninstalling setuptools-65.5.0:
      Successfully uninstalled setuptools-65.5.0
Successfully installed build-0.9.0 click-8.1.3 packaging-22.0 pep517-0.13.0 pip-tools-6.11.0 setuptools-65.6.3 wheel-0.38.4
WARNING: using legacy resolver is deprecated and will be removed in future versions. The default resolver will be change to 'backtracking' in 7.0.0 version. Specify --resolver=backtracking to silence this warning.
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --output-file=requirements.txt requirements.in
#
-e file:.#egg=xyz
    # via -r requirements.in
attrs==22.1.0
    # via
    #   automat
    #   twisted
automat==22.10.0
    # via twisted
constantly==15.1.0
    # via twisted
hyperlink==21.0.0
    # via twisted
idna==3.4
    # via hyperlink
incremental==22.10.0
    # via twisted
pyhamcrest==2.0.4
    # via twisted
six==1.16.0
    # via automat
twisted==19.7.0 ; platform_system != "Darwin" and platform_system != "Windows"
    # via
    #   -r requirements.in
    #   xyz
zope-interface==5.5.2
    # via twisted

# The following packages are considered to be unsafe in a requirements file:
# setuptools