Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Poetry 1.4.0 can't build (editable) dependencies requiring setuptools <64 #7583

Closed
4 tasks done
JacobHayes opened this issue Mar 1, 2023 · 25 comments
Closed
4 tasks done
Labels
area/installer Related to the dependency installer kind/bug Something isn't working as expected status/triage This issue needs to be triaged

Comments

@JacobHayes
Copy link

  • Poetry version: Poetry (version 1.4.0)

  • Python version: 3.11.2

  • OS version and name: macOS 13.2.1

  • pyproject.toml: https://gist.github.com/JacobHayes/110054b6c6d19ca9d6d811c8b4eca374

  • I am on the latest stable Poetry version, installed using a recommended method.

  • I have searched the issues of this repo and believe that this is not a duplicate.

  • I have consulted the FAQ and blog for any relevant entries or release notes.

  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

After upgrading to poetry 1.4.0, I get errors installing local editable deps that require setuptools <64. setuptools 64 changed how editable deps are installed, breaking static analysis tools. It's limited to some perhaps esoteric cases (namespace packages where you don't want to include the namespace itself... which you usually don't).

The error is Backend operation failed: HookMissing('build_editable'), which presumably was added in setuptools 64.

In the grand scheme of setuptools versions, 64 is relatively new (Aug 2022) and think I've occasionally seen much older pins. That said, I don't particularly care if poetry supports the older setuptools versions (I'm slowly migrating away from a structure causing this issue anyway), but it would be good to have the setuptools>64 requirement for built packages documented (at least in the CHANGELOGs) and report a better error, if not some installer fallback.


Here's the sanitized output:

$ poetry install
Installing dependencies from lock file

Package operations: 0 installs, 16 updates, 0 removals

  • Updating my-package (7.1.0 /Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package/src -> 7.1.0 /Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package): Failed

  ChefBuildError

  Backend operation failed: HookMissing('build_editable')

  at ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/installation/chef.py:152 in _prepare
      148│
      149│                 error = ChefBuildError("\n\n".join(message_parts))
      150│
      151│             if error is not None:
    → 152│                 raise error from None
      153│
      154│             return path
      155│
      156│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with my-package (7.1.0 /Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "my-package @ file:///Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package ; python_version >= "3.7""'.

[...] # Same for other editable installs

Trying the command (which was improperly quoted with the python_version) to check whether it is a poetry or build issue succeeds:

$ pip wheel --use-pep517 'my-package @ file:///Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package ; python_version >= "3.7"'
Processing ./packages/my-package
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting hera-workflows==4.3.0
  Using cached hera_workflows-4.3.0-py3-none-any.whl (75 kB)
Collecting argo-workflows==6.3.5
  Using cached argo_workflows-6.3.5-py3-none-any.whl (1.3 MB)
Collecting pytz>=2021.3
  Using cached pytz-2022.7.1-py2.py3-none-any.whl (499 kB)
Collecting python-dateutil>=2.8.2
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting certifi>=2021.10.8
  File was already downloaded /Users/jacobhayes/src/github.com/my-org/my-repo/certifi-2022.12.7-py3-none-any.whl
Collecting urllib3>=1.26.8
  File was already downloaded /Users/jacobhayes/src/github.com/my-org/my-repo/urllib3-1.26.14-py2.py3-none-any.whl
Collecting six>=1.5
  File was already downloaded /Users/jacobhayes/src/github.com/my-org/my-repo/six-1.16.0-py2.py3-none-any.whl
Saved ./hera_workflows-4.3.0-py3-none-any.whl
Saved ./argo_workflows-6.3.5-py3-none-any.whl
Saved ./python_dateutil-2.8.2-py2.py3-none-any.whl
Saved ./pytz-2022.7.1-py2.py3-none-any.whl
Building wheels for collected packages: my-package
  Building wheel for my-package (pyproject.toml) ... done
  Created wheel for my-package: filename=my_package-7.1.0-py3-none-any.whl size=30495 sha256=4c15347e00c6f017e826918506b167142731600744cc772bd1c704347ba03ce9
  Stored in directory: /Users/jacobhayes/Library/Caches/pip/wheels/a1/2a/10/958efd559fdf9deed798ca0036f61354b4d7e1cb46e8aab7b8
Successfully built my-package
@JacobHayes JacobHayes added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Mar 1, 2023
@dimbleby
Copy link
Contributor

dimbleby commented Mar 1, 2023

related to #7574? not sure I understand how all the parts fit together but I'd try with that fix anyway

@JacobHayes
Copy link
Author

Ah yeah, I forgot to post, but I tried installing from that PR and got the same error.

@dimbleby
Copy link
Contributor

dimbleby commented Mar 1, 2023

I'm not sure how convincing the above pip command is for editable installs: pip install -e ... would be a better test of how pip treats an editable install.

I know that pip has an error message about this, perhaps you ought to run into that .

ERROR: Project <foo> has a 'pyproject.toml' and its build 
backend is missing the 'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', 
it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.

IIUC I think that the dependency on a recent setuptools is something that your project ought to be declaring, rather than a poetry issue?

@radoering
Copy link
Member

HookMissing('build_editable') probably means that old setuptools does not support PEP 660. Not sure what to do about that.

Apart from that the proposed command has to be improved:

  • To avoid the improper quoting, I assume we can strip markers from the dependency because they make no difference.
  • editable is not considered in the proposed command

@JacobHayes Can you try if pip wheel --use-pep517 --editable /Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package fails as expected?

@JacobHayes
Copy link
Author

That works fine:

$ pip wheel --use-pep517 --editable /Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package/
Obtaining file:///Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: my-package
  Building wheel for my-package (pyproject.toml) ... done
  Created wheel for my-package: filename=my_package-0.30.0-py3-none-any.whl size=24771 sha256=add6ceb708edbf59043b593adb537cca5757962ea74d28f43c048993f80c9882
  Stored in directory: /private/var/folders/ss/tgmzwbx52y35qpr7rzrtv8ym0000gn/T/pip-ephem-wheel-cache-pyqynsnp/wheels/d6/09/e4/9af7ef3079cdc2eca788e4b806f2154480d17335d9b58e27d5
Successfully built my-package

I thought even before poetry 1.4.0, these editable deps were being built with --use-pep517. 🤷


@dimbleby

IIUC I think that the dependency on a recent setuptools is something that your project ought to be declaring, rather than a poetry issue?

I think the issue is that my sub-dependency (not the current project) requires an older version of setuptools, which is pinned in the sub-dependency's pyproject.toml

@JacobHayes
Copy link
Author

JacobHayes commented Mar 1, 2023

Also, I think I tried this with #7579, but perhaps I messed up the installation or was still pointing to the wrong installation. I can test again after that is merged.

@dimbleby
Copy link
Contributor

dimbleby commented Mar 1, 2023

so I think pip makes this work by calling setup.py develop - which they will stop doing per pypa/pip#11457 (but perhaps not soon)

poetry is often at the vanguard of dropping support for such things (or less willing to sink effort into backwards compatibility, depending on your point of view). eg see also its insistence on pep-517 builds

hard to say as yet how common it is for projects deliberately to pin to an old version of setuptools. it feels like an unusual thing to do; on the other hand you've raised this barely a day after poetry 1.4.0 was released so perhaps there are more like you out there.

you say that you don't particularly care about support for older setuptools and I tend to think that we should encourage that position. ie I'd advocate for poetry telling people who run into this: you need to allow a newer setuptools (or fall back to the old installer, or just not use poetry).

dunno whether this is worth a new error message specifically calling out this possible cause, perhaps it depends how many duplicates of this issue show up!

Regardless, thanks for the report, it has sent me down an interesting technical rabbit-hole...

@radoering radoering added the area/installer Related to the dependency installer label Mar 1, 2023
@basti8909
Copy link

basti8909 commented Mar 2, 2023

We ran into the same issue that @JacobHayes mentioned since we updated to Poetry 1.4.0 with an editable dependency:

[tool.poetry.dependencies]
shared = {path = "../shared", develop = true}

Poetry install fails with:

Installing shared (0.0.3 /builds/platform/shared)
  ChefBuildError
  Backend operation failed: HookMissing('build_editable')
  at /usr/local/lib/python3.8/site-packages/poetry/installation/chef.py:152 in _prepare
      148│ 
      149│                 error = ChefBuildError("\n\n".join(message_parts))
      150│ 
      151│             if error is not None:
    → 152│                 raise error from None
      153│ 
      154│             return path
      155│ 
      156│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
Note: This error originates from the build backend, and is likely not a problem with poetry but with shared (0.0.3 /builds/platform/shared) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "shared @ file:///builds/platform/shared ; python_version >= "3.8" and python_version < "4.0""'.

The check for pep517 succeeds:

$ pip wheel --use-pep517 "shared @ file:///builds/platform/shared"
Processing /builds/platform/shared
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
...
...
...
Saved ./setuptools-67.4.0-py3-none-any.whl
Building wheels for collected packages: shared
  Building wheel for shared (pyproject.toml) ... done
  Created wheel for shared: filename=shared-0.0.3-py3-none-any.whl size=57753 sha256=b191d7f34830f93955e6c7cb3e48804755988fb5f92672d5c8188f48867bf98f
  Stored in directory: /tmp/pip-ephem-wheel-cache-1u8wosvq/wheels/22/cc/8a/beebc660a7666f45746f26a1401c4cb6ed777b35c62efb5744
Successfully built shared

For now we went back to Poetry 1.3.2

Edit
Forgot to mention: We use setuptools >67 in all projects so here it can not be related to a pinned older version in a sub-dependency.

@dimbleby
Copy link
Contributor

dimbleby commented Mar 2, 2023

please provide a reproducible example if you hope for anyone to look at this

@miguelvalente
Copy link

miguelvalente commented Mar 2, 2023

Wrong Tab

@adinhodovic
Copy link

We are generating an OpenAPI client(openapi-generator) and testing it in CI using editable mode. With 1.4, the client is not importable:

<project>/core/tests/test_client.py:15:0: E0401: Unable to import 'openapi_client.apis' (import-error)
<project>/core/tests/test_client.py:15:0: E0611: No name 'apis' in module 'openapi_client' (no-name-in-module)
<project>/core/tests/test_client.py:27:0: E0401: Unable to import 'openapi_client.models' (import-error)
etc.

Reverting to 1.3.1 worked

@dimbleby
Copy link
Contributor

dimbleby commented Mar 2, 2023

@miguelvalente you have replied to a comment from a different issue altogether, I see no connection with this issue

@adinhodovic: what you are describing also has no apparent connection with what this issue is about

@md384
Copy link
Contributor

md384 commented Mar 3, 2023

Can reproduce with a simple example of two projects (lib-a and lib-b),

[tool.poetry]
name = "lib-a"
version = "0.1.0"
description = ""
authors = ["John Smith <john@example.com>"]
packages = [{include = "lib_a"}]

[tool.poetry.dependencies]
python = "~3.9"
requests = "^2.21.0"

[tool.poetry.group.dev.dependencies]
pytest = "^7.2.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
[tool.poetry]
name = "lib-b"
version = "0.1.0"
description = ""
authors = ["John Smith <john@example.com>"]
packages = [{include = "lib_b"}]

[tool.poetry.dependencies]
python = "~3.9"
requests = "^2.21.0"
lib-a = {path = "../libA", develop = true}

[tool.poetry.group.dev.dependencies]
pytest = "^7.2.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Throws

  • Installing lib-a (0.1.0 /Users/matt/src/pythonProject/lib/libA): Failed

  ChefBuildError

  Backend operation failed: HookMissing('build_editable')

  at ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/installation/chef.py:152 in _prepare
      148│ 
      149│                 error = ChefBuildError("\n\n".join(message_parts))
      150│ 
      151│             if error is not None:
    → 152│                 raise error from None
      153│ 
      154│             return path
      155│ 
      156│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with lib-a (0.1.0 /Users/matt/src/pythonProject/lib/libA) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "lib-a @ file:///Users/matt/src/pythonProject/lib/libA ; python_version >= "3.9" and python_version < "3.10""'.

when running poetry install on lib-b

@dimbleby
Copy link
Contributor

dimbleby commented Mar 3, 2023

@md384 you are using a (different) ancient backend

you want something like

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

@md384
Copy link
Contributor

md384 commented Mar 3, 2023

@dimbleby yes, that was the problem for us, changing that in our pyproject.toml fixed our problems.

@yeus
Copy link

yeus commented Mar 30, 2023

We ran into the same issue that @JacobHayes mentioned since we updated to Poetry 1.4.0 with an editable dependency:

[tool.poetry.dependencies]
shared = {path = "../shared", develop = true}

we have the same problem, also using the "develop = true" flag in on of our dependencies. Also going back to 1.3.

@schristensen-unhaze
Copy link

@md384 you are using a (different) ancient backend

you want something like

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

After upgrading to poetry 1.4.2 we had an issue where setuptools was finding multiple top level packages, and changing the build-system to the above fixed our issue. This is a huge relief because it means that we don't have to change our pyproject.toml files to follow pep 621 just yet.

thrix added a commit to testing-farm/gluetool-modules that referenced this issue Jun 5, 2023
I got struck by this problem:

   python-poetry/poetry#7583

While trying to install `gluetool-modules-internal` with path
dependency on `gluetool-modules-framework` with `develop = true`.

```
  • Installing gluetool-modules-framework (0.9.0 /var/home/mvadkert/git/gitlab.com/testing-farm/gluetool-modules): Failed

  ChefBuildError

  Backend operation failed: HookMissing('build_editable')
```

This resolves the problem.

Signed-off-by: Miroslav Vadkerti <mvadkert@redhat.com>
@thrix
Copy link

thrix commented Jun 5, 2023

Yeah, seems poetry init nowadays creates:

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

thrix added a commit to testing-farm/gluetool-modules that referenced this issue Jun 7, 2023
I got struck by this problem:

   python-poetry/poetry#7583

While trying to install `gluetool-modules-internal` with path
dependency on `gluetool-modules-framework` with `develop = true`.

```
  • Installing gluetool-modules-framework (0.9.0 /var/home/mvadkert/git/gitlab.com/testing-farm/gluetool-modules): Failed

  ChefBuildError

  Backend operation failed: HookMissing('build_editable')
```

This resolves the problem.

Signed-off-by: Miroslav Vadkerti <mvadkert@redhat.com>
thrix added a commit to testing-farm/gluetool-modules that referenced this issue Jun 7, 2023
I got struck by this problem:

   python-poetry/poetry#7583

While trying to install `gluetool-modules-internal` with path
dependency on `gluetool-modules-framework` with `develop = true`.

```
  • Installing gluetool-modules-framework (0.9.0 /var/home/mvadkert/git/gitlab.com/testing-farm/gluetool-modules): Failed

  ChefBuildError

  Backend operation failed: HookMissing('build_editable')
```

This resolves the problem.

Signed-off-by: Miroslav Vadkerti <mvadkert@redhat.com>
thrix added a commit to testing-farm/gluetool-modules that referenced this issue Jun 7, 2023
I got struck by this problem:

   python-poetry/poetry#7583

While trying to install `gluetool-modules-internal` with path
dependency on `gluetool-modules-framework` with `develop = true`.

```
  • Installing gluetool-modules-framework (0.9.0 /var/home/mvadkert/git/gitlab.com/testing-farm/gluetool-modules): Failed

  ChefBuildError

  Backend operation failed: HookMissing('build_editable')
```

This resolves the problem.

Signed-off-by: Miroslav Vadkerti <mvadkert@redhat.com>
thrix added a commit to testing-farm/gluetool-modules that referenced this issue Jun 7, 2023
I got struck by this problem:

   python-poetry/poetry#7583

While trying to install `gluetool-modules-internal` with path
dependency on `gluetool-modules-framework` with `develop = true`.

```
  • Installing gluetool-modules-framework (0.9.0 /var/home/mvadkert/git/gitlab.com/testing-farm/gluetool-modules): Failed

  ChefBuildError

  Backend operation failed: HookMissing('build_editable')
```

This resolves the problem.

Signed-off-by: Miroslav Vadkerti <mvadkert@redhat.com>
@s3rgeym

This comment was marked as spam.

@SandersAaronD
Copy link

This is still happening, although the circumstances for it are admittedly weird.

@franciscoalves
Copy link

This is for anyone that uses Homebrew on Mac having this issue. I've installed poetry with homebrew and I don't recommend it because over time running brew upgrades created a version mistmatch between the python formula and the poetry formula and I had python 3.11 with a poetry that used the forumla for python 3.12.
TLDR:
This is what fixed the issue for me:

  1. install the python you want with homebrew.
    brew install python@3.11
  2. use pip that came with that python to install pipx.
    pip3.11 install pipx
  3. use pipx to install poetry
    pipx install poetry

Now poetry will be tied to that python version you chose and those errors will go away.

jacobwegner added a commit to jacobwegner/strawberry-django-auth that referenced this issue Jan 8, 2024
refs https://python-poetry.org/docs/pyproject#poetry-and-pep-517

Fixes an error encountered when trying to add a local version of this package as an editable dependency to another local project:

```
poetry add -e <path-to-repo>
...

 ChefBuildError

  Backend operation failed: HookMissing('build_editable')

  at ~/.pyenv/versions/3.11.4/lib/python3.11/site-packages/poetry/installation/chef.py:147 in _prepare
      143│
      144│                 error = ChefBuildError("\n\n".join(message_parts))
      145│
      146│             if error is not None:
    → 147│                 raise error from None
      148│
      149│             return path
      150│
      151│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with strawberry-django-auth (0.376.4 ~/repos/strawberry-django-auth) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 --editable "/Users/jwegner/Data/development/repos/jacobwegner/strawberry-django-auth"'.
```

refs python-poetry/poetry#7583 (comment)
@rmarscher
Copy link

Sorry if this is not helpful... but I had the same error and eventually discovered it was because the python version I had built via pyenv was missing libffi. I added libffi-devel to my system and reinstalled python and then poetry was able to install the dependencies without an issue.

@dimbleby
Copy link
Contributor

pretty much all comments on this are unrelated to the thing that it actually reports, which is a very specific issue about editable dependencies and pins to old versions of setuptools.

setuptools 64.0 was not that old when this was reported: but it is now a year older - the likelihood and value of anyone doing anything with this are both diminishing by the day.

suggest that the issue might as well be be closed out, it seems to be acting only as a honeytrap

@JacobHayes
Copy link
Author

JacobHayes commented Mar 12, 2024

Thanks, forgot this issue was still open. Agreed on diminishing returns to OP

@JacobHayes JacobHayes closed this as not planned Won't fix, can't repro, duplicate, stale Mar 12, 2024
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 12, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area/installer Related to the dependency installer kind/bug Something isn't working as expected status/triage This issue needs to be triaged
Projects
None yet
Development

No branches or pull requests