PEP 517 - can't build a dependency without setup.py #6181

AvnerCohen · 2019-01-23T12:31:24Z

Environment

pip version: 19.0
Python version: 2.7.15
OS: OSx

Description
Deploying a dependency that does not have setup.py, but does have pyproject.toml, fails with "setup.py" not found

  Complete output from command /Users/[username]/.virtualenvs/screenit/bin/python2.7 /Users/[username]/.virtualenvs/screenit/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/rn/h895ldtn7rd_6gc9mgjdb6zh0000gn/T/tmpGaZn3F:
  Traceback (most recent call last):
    File "/Users/[username]/.virtualenvs/screenit/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
      main()
    File "/Users/[username]/.virtualenvs/screenit/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/[username]/.virtualenvs/screenit/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 54, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/rn/h895ldtn7rd_6gc9mgjdb6zh0000gn/T/pip-build-env-pEtv1_/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 115, in get_requires_for_build_wheel
      return _get_build_requires(config_settings, requirements=['wheel'])
    File "/private/var/folders/rn/h895ldtn7rd_6gc9mgjdb6zh0000gn/T/pip-build-env-pEtv1_/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 101, in _get_build_requires
      _run_setup()
    File "/private/var/folders/rn/h895ldtn7rd_6gc9mgjdb6zh0000gn/T/pip-build-env-pEtv1_/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 82, in _run_setup
      f = getattr(tokenize, 'open', open)(__file__)
  IOError: [Errno 2] No such file or directory: 'setup.py'

Expected behavior
Build is expected to complete succesfully.

How to Reproduce
pip install anypackage.with.toml.and.no.setup.py

Output

  Complete output from command /Users/[username]/.virtualenvs/screenit/bin/python2.7 /Users/[username]/.virtualenvs/screenit/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/rn/h895ldtn7rd_6gc9mgjdb6zh0000gn/T/tmpGaZn3F:
  Traceback (most recent call last):
    File "/Users/[username]/.virtualenvs/screenit/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
      main()
    File "/Users/[username]/.virtualenvs/screenit/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/[username]/.virtualenvs/screenit/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 54, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/rn/h895ldtn7rd_6gc9mgjdb6zh0000gn/T/pip-build-env-pEtv1_/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 115, in get_requires_for_build_wheel
      return _get_build_requires(config_settings, requirements=['wheel'])
    File "/private/var/folders/rn/h895ldtn7rd_6gc9mgjdb6zh0000gn/T/pip-build-env-pEtv1_/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 101, in _get_build_requires
      _run_setup()
    File "/private/var/folders/rn/h895ldtn7rd_6gc9mgjdb6zh0000gn/T/pip-build-env-pEtv1_/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 82, in _run_setup
      f = getattr(tokenize, 'open', open)(__file__)
  IOError: [Errno 2] No such file or directory: 'setup.py'

The text was updated successfully, but these errors were encountered:

pradyunsg · 2019-01-23T12:32:32Z

Seems like #6170. Asking the same questions here:

Could you show us a minimal example of this behavior? Does pyproject.toml not specify a build system and setup.py doesn't exist?

AvnerCohen · 2019-01-23T12:43:00Z

Here is a sample that allows recreating this:

pip install "git+ssh://git@github.com/AvnerCohen/pytoml_sample.git@master#egg=pytoml_sample"

pfmoore · 2019-01-23T12:52:03Z

You haven't specified the build backend in pyproject.toml. So pip expectes a setup.py as it will default to setuptools.

AvnerCohen · 2019-01-23T13:14:39Z

Thanks @pfmoore this is defiantly progress, and missout on my end, I was actually thinking the ability to install packages from pyproject.toml is being supported, but I guess what is supported is the ability to hand over the build system to another tool, or something.

Based on that, I have now added:

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

This now fails with:

  Collecting pendulum<3.0,>=2.0
  Collecting glom<19.0,>=18.3
    Downloading https://files.pythonhosted.org/packages/73/81/e5873a30b4ef7eab34a13997d2ad90d81231bba332bbf151df905b4c0576/glom-18.4.0-py2-none-any.whl (44kB)
  Collecting python-dateutil<3.0,>=2.7
    Using cached https://files.pythonhosted.org/packages/74/68/d87d9b36af36f44254a8d512cbfc48369103a3b9e474be9bdfe536abfc45/python_dateutil-2.7.5-py2.py3-none-any.whl
  Collecting odata
    Could not find a version that satisfies the requirement odata (from versions: )
  No matching distribution found for odata

  ----------------------------------------
Command "/Users/[username]/.virtualenvs/screenit/bin/python2.7 /Users/[username]/.virtualenvs/screenit/lib/python2.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/rn/h895ldtn7rd_6gc9mgjdb6zh0000gn/T/pip-build-env-7WNOiI/normal --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- pendulum>=2.0,<3.0 glom>=18.3,<19.0 python-dateutil>=2.7,<3.0 odata toolz>=0.9.0,<0.10.0 formation>=0.1.27,<0.2.0 attr>=0.3.1,<0.4.0 PyJWT>=1.7,<2.0 structlog>=18.2,<19.0 attrs-serde>=0.1.1,<0.2.0 cytoolz>=0.9.0,<0.10.0 xmltodict>=0.11.0,<0.12.0" failed with error code 1 in None

Specifically - No matching distribution found for odata

Even though, when running poetry install on the tafrget lib it works just fine and installs:

...
- Installing odata (0.3 f2afea8)
..

E.g. something in the handoff to poetry does not work as expected, note that this specific lib is installed from git and not fro pypi:

https://github.com/AvnerCohen/pytoml_sample/blob/master/pyproject.toml#L25

pfmoore · 2019-01-23T13:21:56Z

I'd look at what the poetry PEP 517 backend is doing. You may need to somehow enable debug output from that backend, but how you'd do that is backend-specific.

pfmoore · 2019-01-23T13:27:27Z

Persumably if you try to install odata by hand it works?

dstufft · 2019-01-23T13:35:42Z

I think, this is effectively a poetry bug I think. Looking at the pyproject.toml, it has:

odata = {git = "https://github.com/tuomur/python-odata"}

When I slightly tweak my local copy of pip to spew out the exact information that the poetry build backend returned for what the requirements are, I see:

['toolz>=0.9.0,<0.10.0',
 'cytoolz>=0.9.0,<0.10.0',
 'requests>=2.20,<3.0',
 'xmltodict>=0.11.0,<0.12.0',
 'python-dateutil>=2.7,<3.0',
 'odata',
 'glom>=18.3,<19.0',
 'pendulum>=2.0,<3.0',
 'attrs-serde>=0.1.1,<0.2.0',
 'structlog>=18.2,<19.0',
 'formation>=0.1.27,<0.2.0',
 'PyJWT>=1.7,<2.0',
 'attr>=0.3.1,<0.4.0']

Since odata does not exist on PyPI, and poetry is omitting information about where to locate odata, we are not able to install odata to setup the build environment.

When I tell pip where to find odata at, it works (well it still fails, but it fails because of a poetry issue with your pyproject.toml).

Try using:

pip install . --find-links 'git+https://github.com/tuomur/python-odata#egg=odata-0.3'

There's nothing actionable here for pip, poetry needs to correctly handle git dependencies.

AvnerCohen · 2019-01-23T13:44:59Z

@dstufft I could still be missing something, but I think this is too hasty to close this.

Installing the dependency directly from poetry, as I wrote above works just fine:

(pytoml_sample) pytoml_sample/ on master › poetry install
Installing dependencies from lock file


Package operations: 78 installs, 0 updates, 0 removals

  - Installing certifi (2018.11.29)
  - Installing chardet (3.0.4)
  - Installing idna (2.8)
  - Installing six (1.12.0)
  - Installing urllib3 (1.22)
  - Installing attrs (18.2.0)
  - Installing lockfile (0.12.2)
  - Installing msgpack (0.6.0)
  - Installing multidict (3.3.1)
  - Installing pastel (0.1.0)
  - Installing pylev (1.3.0)
  - Installing pyrsistent (0.14.9)
  - Installing requests (2.21.0)
  - Installing webencodings (0.5.1)
  - Installing argh (0.26.2)
  - Installing atomicwrites (1.2.1)
  - Installing boltons (18.0.1)
  - Installing cachecontrol (0.12.5)
  - Installing cachy (0.2.0)
  - Installing cleo (0.6.8)
  - Installing colorama (0.4.1)
  - Installing html5lib (1.0.1)
  - Installing jsonschema (3.0.0a4)
  - Installing markupsafe (1.1.0)
  - Installing more-itertools (4.3.0)
  - Installing pathtools (0.1.2)
  - Installing pkginfo (1.5.0.1)
  - Installing pluggy (0.8.0)
  - Installing py (1.7.0)
  - Installing pycodestyle (2.4.0)
  - Installing pyparsing (2.3.1)
  - Installing pyyaml (3.13)
  - Installing requests-toolbelt (0.8.0)
  - Installing shellingham (1.2.8)
  - Installing tomlkit (0.5.3)
  - Installing toolz (0.9.0)
  - Installing wrapt (1.10.11)
  - Installing yarl (0.18.0)
  - Installing autopep8 (1.4.3)
  - Installing click (7.0)
  - Installing coverage (4.4.2)
  - Installing cytoolz (0.9.0.1)
  - Installing docopt (0.6.2)
  - Installing face (0.1.0)
  - Installing filelock (3.0.10)
  - Installing jinja2 (2.10)
  - Installing pbr (5.1.1)
  - Installing poetry (0.12.11)
  - Installing py-cpuinfo (4.0.0)
  - Installing pybreaker (0.4.5)
  - Installing pytest (3.10.1)
  - Installing python-dateutil (2.7.5)
  - Installing pytzdata (2018.7)
  - Installing termcolor (1.1.0)
  - Installing toml (0.10.0)
  - Installing vcrpy (2.0.1)
  - Installing virtualenv (16.1.0)
  - Installing watchdog (0.9.0)
  - Installing yapf (0.25.0)
  - Installing attr (0.3.1)
  - Installing attrs-serde (0.1.1)
  - Installing changelog-cli (0.6.2)
  - Installing formation (0.1.27)
  - Installing glom (18.3.1)
  - Installing mock (2.0.0)
  - Installing odata (0.3 f2afea8)
  - Installing pendulum (2.0.4)
  - Installing poetry-setup (0.3.6)
  - Installing pyjwt (1.7.1)
  - Installing pytest-benchmark (3.1.1)
  - Installing pytest-testmon (0.9.14)
  - Installing pytest-vcr (1.0.1)
  - Installing pytest-watch (4.2.0)
  - Installing scrub (0.1.2)
  - Installing snapshottest (0.5.0)
  - Installing structlog (18.2.0)
  - Installing tox (3.5.3)
  - Installing xmltodict (0.11.0)

You can also see the poetry lock file that is created as expected and clearly pointing to the git repo:
https://github.com/AvnerCohen/pytoml_sample/blob/master/pyproject.lock#L247

pfmoore · 2019-01-23T13:48:13Z

The poetry backend has to tell pip the requirement (in a form that pip can use). Pip uses that to set up the build environment. That's what get_requires_for_build_wheel is for. The problem here is that the poetry backend isn't doing this. So that's why we're saying this is a poetry backend problem.

Note - this is likely the first time the poetry backend has ever seen real-world use, so it's to be expected that it would have teething troubles.

dstufft · 2019-01-23T13:50:25Z

Yes, it works from poetry because poetry is directly reading the [tool.poetry.dependencies] section of the pyproject.toml.

If you notice, that is a section that is entirely poetry specific and pip has no knowledge of it. When pip attempts to build the wheel, it invokes the poetry build backend API and asks it "hey, what dependencies do you need to build a wheel so I can install them for it". The intent here is that poetry (or any build backend) will use their tool specific mechanism for locating those dependencies, and report them back to pip so pip can install them.

Unfortunately, when pip asks poetry for the build dependencies, it returns just "odata" without any additional information on where to locate that dependency from. Since we don't have any additional information, we end up searching the configured repositories for it, which defaults to PyPI (and the command I linked adds another location, the git URL). When pip can't find the dependency in it's configured locations, it appropriately fails.

As an aside, It seems odd to me that poetry is returning the runtime dependencies when we ask it what the build dependencies are, but that may just be a design decision that poetry made.

AvnerCohen · 2019-01-23T13:57:08Z

This is great stuff guys, I appreciate the help and detailed responses.
I now understand the issue might be specifically in the backend API impl of poerty.

I'll head over there to try and get it resolved, I guess a better python world is when everything will play nicely together.

Thanks!

EDITED:
I see this is already reported - python-poetry/poetry#825

lock · 2019-05-29T07:38:45Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

dstufft closed this as completed Jan 23, 2019

AvnerCohen mentioned this issue Jan 23, 2019

Git dependency issues with pip 19 python-poetry/poetry#825

Closed

3 tasks

kipyin mentioned this issue Feb 4, 2019

Git requirements are not present in setup.py python-poetry/poetry#443

Closed

3 tasks

lock bot added the auto-locked Outdated issues that have been locked by automation label May 29, 2019

lock bot locked as resolved and limited conversation to collaborators May 29, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PEP 517 - can't build a dependency without setup.py #6181

PEP 517 - can't build a dependency without setup.py #6181

AvnerCohen commented Jan 23, 2019

pradyunsg commented Jan 23, 2019

AvnerCohen commented Jan 23, 2019

pfmoore commented Jan 23, 2019

AvnerCohen commented Jan 23, 2019

pfmoore commented Jan 23, 2019

pfmoore commented Jan 23, 2019

dstufft commented Jan 23, 2019

AvnerCohen commented Jan 23, 2019

pfmoore commented Jan 23, 2019

dstufft commented Jan 23, 2019

AvnerCohen commented Jan 23, 2019 •

edited

Loading

lock bot commented May 29, 2019

PEP 517 - can't build a dependency without setup.py #6181

PEP 517 - can't build a dependency without setup.py #6181

Comments

AvnerCohen commented Jan 23, 2019

pradyunsg commented Jan 23, 2019

AvnerCohen commented Jan 23, 2019

pfmoore commented Jan 23, 2019

AvnerCohen commented Jan 23, 2019

pfmoore commented Jan 23, 2019

pfmoore commented Jan 23, 2019

dstufft commented Jan 23, 2019

AvnerCohen commented Jan 23, 2019

pfmoore commented Jan 23, 2019

dstufft commented Jan 23, 2019

AvnerCohen commented Jan 23, 2019 • edited Loading

lock bot commented May 29, 2019

AvnerCohen commented Jan 23, 2019 •

edited

Loading