Exception when adding apache-beam[gcp] to empty project

[guyzyl]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 12.0.1
• Poetry version: 1.1.12, 1.2.0a2 (tried both)
• Link of a Gist with the contents of your pyproject.toml file: pyproject.toml

Issue

I'm receiving an exception when trying to add apache-beam[gcp] to a new and empty project.
It is possible that the issue resides with the library itself and not Poetry. If that's the case I'll make sure to open a ticket for them instead :)

The issue can easily be replicated with Poetry by creating a new empty folder, and running:

poetry init
poetry add apache-beam[gcp]

This is the full exception I received:

 • Installing apache-beam (2.34.0): Failed

  CalledProcessError

  Command '['/Users/guyzyl/sightsource/test/.venv/bin/python', '/usr/local/lib/python3.9/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/Users/guyzyl/sightsource/test/.venv', '--no-deps', 'file:/Users/guyzyl/Library/Caches/pypoetry/artifacts/a6/78/23/9338a4b21c37427c5b18dfbd31d03b7b5240a24aa9d82bc6182708fb9d/apache-beam-2.34.0.zip']' returned non-zero exit status 1.

  at /usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py:528 in run
       524│             # We don't call process.wait() as .__exit__ does that for us.
       525│             raise
       526│         retcode = process.poll()
       527│         if check and retcode:
    →  528│             raise CalledProcessError(retcode, process.args,
       529│                                      output=stdout, stderr=stderr)
       530│     return CompletedProcess(process.args, retcode, stdout, stderr)
       531│
       532│

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/Users/guyzyl/sightsource/test/.venv/bin/python', '/usr/local/lib/python3.9/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/Users/guyzyl/sightsource/test/.venv', '--no-deps', 'file:/Users/guyzyl/Library/Caches/pypoetry/artifacts/a6/78/23/9338a4b21c37427c5b18dfbd31d03b7b5240a24aa9d82bc6182708fb9d/apache-beam-2.34.0.zip'] errored with the following return code 1, and output:
  Processing /Users/guyzyl/Library/Caches/pypoetry/artifacts/a6/78/23/9338a4b21c37427c5b18dfbd31d03b7b5240a24aa9d82bc6182708fb9d/apache-beam-2.34.0.zip
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Users/guyzyl/sightsource/test/.venv/bin/python /var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpj7fug9l2_in_process.py get_requires_for_build_wheel /var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpyf7pddkh
         cwd: /private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-req-build-vua9jsg4
    Complete output (26 lines):
    Traceback (most recent call last):
      File "/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpj7fug9l2_in_process.py", line 280, in <module>
        main()
      File "/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpj7fug9l2_in_process.py", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpj7fug9l2_in_process.py", line 114, in get_requires_for_build_wheel
        return hook(config_settings)
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
        return self._get_build_requires(
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 143, in _get_build_requires
        self.run_setup()
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 267, in run_setup
        super(_BuildMetaLegacyBackend,
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 158, in run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 97, in <module>
        _PIP_VERSION = get_distribution('pip').version
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/pkg_resources/__init__.py", line 471, in get_distribution
        dist = get_provider(dist)
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/pkg_resources/__init__.py", line 347, in get_provider
        return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/pkg_resources/__init__.py", line 891, in require
        needed = self.resolve(parse_requirements(requirements))
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/pkg_resources/__init__.py", line 777, in resolve
        raise DistributionNotFound(req, requirers)
    pkg_resources.DistributionNotFound: The 'pip' distribution was not found and is required by the application
    ----------------------------------------
  WARNING: Discarding file:///Users/guyzyl/Library/Caches/pypoetry/artifacts/a6/78/23/9338a4b21c37427c5b18dfbd31d03b7b5240a24aa9d82bc6182708fb9d/apache-beam-2.34.0.zip. Command errored out with exit status 1: /Users/guyzyl/sightsource/test/.venv/bin/python /var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpj7fug9l2_in_process.py get_requires_for_build_wheel /var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpyf7pddkh Check the logs for full command output.
  ERROR: Command errored out with exit status 1: /Users/guyzyl/sightsource/test/.venv/bin/python /var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpj7fug9l2_in_process.py get_requires_for_build_wheel /var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpyf7pddkh Check the logs for full command output.


  at /usr/local/lib/python3.9/site-packages/poetry/utils/env.py:1300 in _run
      1296│                 output = subprocess.check_output(
      1297│                     cmd, stderr=subprocess.STDOUT, env=env, **kwargs
      1298│                 )
      1299│         except CalledProcessError as e:
    → 1300│             raise EnvCommandError(e, input=input_)
      1301│
      1302│         return decode(output)
      1303│
      1304│     def execute(self, bin: str, *args: str, **kwargs: Any) -> Optional[int]:

The following error occurred when trying to handle this error:


  PoetryException

  Failed to install file:/Users/guyzyl/Library/Caches/pypoetry/artifacts/a6/78/23/9338a4b21c37427c5b18dfbd31d03b7b5240a24aa9d82bc6182708fb9d/apache-beam-2.34.0.zip

  at /usr/local/lib/python3.9/site-packages/poetry/utils/pip.py:60 in pip_install
       56│                     *env.get_pip_command(),
       57│                     *args,
       58│                     env={**os.environ, "PYTHONPATH": str(env.purelib)},
       59│                 )
    →  60│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       61│
       62│
       63│ def pip_editable_install(directory: Path, environment: Env) -> Union[int, str]:
       64│     return pip_install(

[RobMcKiernan]

I'm also experiencing this issue. I tried it on a non-empty project as well, and that also fails.

[clintonroy]

Works on linux for me at least. It looks to me like you need pip installed somewhere. Is it available on the host at all?

[RobMcKiernan]

Sorry, should have said- I'm on linux mint (cinnamon 20.2).

My python set up is: I'm always in a conda environment with the python I want to be using, and that has pip installed. Then I use poetry within that conda environment.

I can install apache-beam[GCP] using pip in the conda environment, but that means that poetry isn't being used to manage my package dependencies, which of course defeats the object of using poetry

[guyzyl]

That's the issue. Works when directly installing via Pip, but not when done through Poetry.
@clintonroy I have pip installed on my machine, v21.3.1.

[RobMcKiernan]

Just fixed this on my machine :D

My issue was that I had a mess of poetry installations, through partly installing them through pip, and partly through the installation script.

My solution was to remove all poetry installations, clean up my conda environments and ensure it only uses pip. Then to tell poetry to not use the installer:

# remove all your poetrys
# Repeat these for every conda environment (I actually just deleted all my conda environments except (base), and did these steps just for (base))
python -m pip uninstall poetry
python -m pip3 uninstall poetry

curl -sSL https://install.python-poetry.org | python3 - --uninstall

# install poetry for each separate conda environment
python -m pip install poetry # I included the --pre flag here, but that's for separate reasons, and probably doesn't matter

# tell poetry to ignore the installer
poetry config experimental.new-installer false

# install apache-beam[gcp]
poetry add "apache-beam[gcp]"

My suggestion would be to start with

which python
which poetry

and go from there

Hope this helps

[guyzyl]

@RobMcKiernan thanks for you solution insight!
I still think that there's an underlying bug with Poetry that should be found and resolved.

[dimbleby]

apache-beam is making a wholly redundant check that a recent version of pip is available in its environment. But

• this need not be true (as in this example, where the pip that is being run is in a different virtual environment altogether)
• the "recent" version being checked for was last updated in 2016, we can reasonably bet that no-one is using an older pip than that.

Suggest that anyone who cares about this issue raise an MR at apache-beam removing this block of code

https://github.com/apache/beam/blob/7ad4864b0cb19b6c8405265f84fff24bf5b2c8b3/sdks/python/setup.py#L93-L101

[oleksandr-yatsuk]

In my case, the issue was related to the experimental installer

poetry config experimental.new-installer false

fixed the issue

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.