[BUG] Isolated build env breaks console scripts installed in other venv (e.g., cmake, ninja)

[XuehaiPan]

Description

For packages that contain extension modules, they may need to use build tools such as cmake / ninja. There are PyPI packages that ship those tools with console scripts. E.g., pip3 install cmake will create an executable ${PROJECT}/venv/bin/cmake.

#!${PROJECT}/venv/bin/python3.13
# -*- coding: utf-8 -*-
import re
import sys
from cmake import cmake
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(cmake())

When installing a user package from the source, pip will create an isolated build environment (venv (B)). During installing the user package, the setup script may invoke the cmake executable in the PATH mentioned above (installed in venv (A)). However, in the build environment (venv (B)), the cmake script (interpreted by Python in venv (A)) failed to find the cmake module installed in venv (A).

See issue scikit-build/cmake-python-distributions#586 for more information.

• [BUG] Cannot find the cmake module in a wheel build venv when the cmake executable is installed in another venv scikit-build/cmake-python-distributions#586

I think this issue is caused by the suspicious sitecustomize.py in the build environment:

pip/src/pip/_internal/build_env.py

Lines 96 to 134 in 93d43c9

	# Customize site to:
	# - ensure .pth files are honored
	# - prevent access to system site packages
	system_sites = _get_system_sitepackages()
	self._site_dir = os.path.join(temp_dir.path, "site")
	if not os.path.exists(self._site_dir):
	os.mkdir(self._site_dir)
	with open(
	os.path.join(self._site_dir, "sitecustomize.py"), "w", encoding="utf-8"
	) as fp:
	fp.write(
	textwrap.dedent(
	"""
	import os, site, sys
	# First, drop system-sites related paths.
	original_sys_path = sys.path[:]
	known_paths = set()
	for path in {system_sites!r}:
	site.addsitedir(path, known_paths=known_paths)
	system_paths = set(
	os.path.normcase(path)
	for path in sys.path[len(original_sys_path):]
	)
	original_sys_path = [
	path for path in original_sys_path
	if os.path.normcase(path) not in system_paths
	]
	sys.path = original_sys_path
	# Second, add lib directories.
	# ensuring .pth file are processed.
	for path in {lib_dirs!r}:
	assert not path in sys.path
	site.addsitedir(path)
	"""
	).format(system_sites=system_sites, lib_dirs=self._lib_dirs)
	)

Expected behavior

The console script (e.g., cmake) should always be runnable if invoked in a subprocess with an absolute path.

pip version

25.0.1

Python version

3.13.2

OS

macOS

How to Reproduce

1. Create a setup.py with content:

# setup.py

import os
import shutil
from pathlib import Path

from setuptools import Extension, setup
from setuptools.command.build_ext import build_ext


HERE = Path(__file__).absolute().parent


class CMakeExtension(Extension):
    def __init__(self, name, source_dir=".", target=None, **kwargs):
        super().__init__(name, sources=[], **kwargs)
        self.source_dir = Path(source_dir).absolute()
        self.target = target if target is not None else name.rpartition(".")[-1]

    @classmethod
    def cmake_executable(cls):
        cmake = os.getenv("CMAKE_EXECUTABLE", "")
        if not cmake:
            cmake = shutil.which("cmake")
        return cmake


class cmake_build_ext(build_ext):
    def build_extension(self, ext):
        if not isinstance(ext, CMakeExtension):
            super().build_extension(ext)
            return

        cmake = ext.cmake_executable()
        if cmake is None:
            raise RuntimeError("Cannot find CMake executable.")

        self.spawn([cmake, "--version"])


setup(
    name="cmake-venv-test",
    version="0.0.1",
    cmdclass={"build_ext": cmake_build_ext},
    ext_modules=[CMakeExtension("cmake_venv_test._C", source_dir=HERE)],
)

2. Run the following commands:

$ python3 -m venv venv
$ source venv/bin/activate
$ pip3 install cmake
$ which cmake
${PROJECT}/venv/bin/cmake

$ pip3 install .
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Processing ${PROJECT}
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: cmake-venv-test
  Building wheel for cmake-venv-test (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for cmake-venv-test (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      running bdist_wheel
      running build
      running build_ext
      ${PROJECT}/venv/bin/cmake --version
      Traceback (most recent call last):
        File "${PROJECT}/venv/bin/cmake", line 5, in <module>
          from cmake import cmake
      ModuleNotFoundError: No module named 'cmake'
      error: command '${PROJECT}/venv/bin/cmake' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for cmake-venv-test
Failed to build cmake-venv-test

ERROR: Failed to build installable wheels for some pyproject.toml based projects (cmake-venv-test)

Output

$ pip3 install .
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Processing ${PROJECT}
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: cmake-venv-test
  Building wheel for cmake-venv-test (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for cmake-venv-test (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      running bdist_wheel
      running build
      running build_ext
      ${PROJECT}/venv/bin/cmake --version
      Traceback (most recent call last):
        File "${PROJECT}/venv/bin/cmake", line 5, in <module>
          from cmake import cmake
      ModuleNotFoundError: No module named 'cmake'
      error: command '${PROJECT}/venv/bin/cmake' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for cmake-venv-test
Failed to build cmake-venv-test

ERROR: Failed to build installable wheels for some pyproject.toml based projects (cmake-venv-test)

The error raises when installing with and without the --editable flag.

There will be no error if do either of the following:

• Add cmake to build-system.requires in pyproject.toml.
• Install with flag --no-build-isolation.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[ichard26]

This is the point of build isolation, the build process shouldn't depend on any Python-level dependencies that are not declared in the build-system.requires field of pyproject.toml. And yes, cmake is a Python-level dependency because pip is the installation method used here

You need to use a pyproject.toml so pip will install everything your package needs to build:

[build-system]
requires = ["setuptools", "cmake"]
build-backend = "setuptools.build_meta"

You already know that though, so I'm not really sure what the problem is here.

[ichard26]

Apparently scikit-build disagrees and believes this is a pip bug. I'll talk with them and return when I understand what's going on here.

[XuehaiPan]

You need to use a pyproject.toml so pip will install everything your package needs to build.
You already know that though, so I'm not really sure what the problem is here.

The cmake PyPI package contains binaries and is not a pure Python package. So the wheel may not be available on some platforms (e.g., FreeBSD) and archs. Users may complain about this if cmake presents in build-system.requires. cc @mgorny

See also:

• [BUG] optree tries to install cmake from PyPI unconditionally metaopt/optree#187 (comment)

[XuehaiPan]

And yes, cmake is a Python-level dependency because pip is the installation method used here

The user's setup script is not trying to import the cmake module. Instead, the cmake executable is invoked in a separate subprocess by build_ext.spawn(...). In the subprocess command line, the absolute path of cmake is given.

[notatallshaw]

I've read through all the links, but I'm still confused as to what the ask is?

If a build time dependency is a Python package it should be in the build-system.requires.

If a build time dependency is an external dependency it should be part of the prerequisite install instructions, pip is bound to Python dependencies, the conda environment has attempted to solve the problem of making all 3rd party dependencies available, but that requires using conda.

If there is some awkward in between where you have to support third party code that's importing things that aren't always there then you're going to have to write some infrastructure to handle that, e.g. a build time dependency that accepts the import but doesn't do anything with it if the external dependency isn't available. It’s messy, but that’s sometimes the problem with supporting arbitrary third party code running in an environment it wasn’t designed for.

Do you have some specific thing pip could do here to alleviate the problem that fits within the spec?

[notatallshaw]

Ah, I think you can ignore my comment, I see @ichard26 has been more deeply investigating the issue.

[ichard26]

I talked over this issue with @henryiii on the PyPA Discord because I realized that I was wholly uninformed about the complexity involved here. My understanding is summarized here:

The problem is that the isolated build logic is messing with the import path of Python subprocesses invoked by the build process that are using the Python from the parent environment (venv A).

• sys.executable of the cmake subprocess is venv/bin/python
• In the build environment, we set PYTHONPATH to include a sitecustomize.py we inject into the isolated build environment
• The injected sitecustomize.py modifies sys.path to isolate any Python subprocesses from the parent environment (aka venv A)

The problem is that even under normal execution, pip's own calls in the build environment (and not whatever the build process may be doing) are also running off the parent Python. In other words, sys.executable=venv/bin/python is always true.

I'm not aware of an easy way to isolate pip's own subprocesses vs other Python subprocesses the build environment may create since the same python executable is being used (venv/bin/python). The calls are identical for all we can tell.

Anyway, as evident by my original response, I am not an expert in build isolation or PEP 517. I apologise for my confusion @XuehaiPan. Thank you for the high-quality bug report even if it flew over my head.

[ichard26]

I'm curious to how uv handles this (since apparently an external cmake does work under uv), but that is far outside my expertise.

[ichard26]

I should clarify that pip doesn't really create a true virtual environment (à la virtualenv/venv). It is indeed an isolated environment, but the isolation is achieved by the site/sys.path manipulations that are also the reason why cmake is broken here. Otherwise, it's really the same Python pip is running off.

[notatallshaw]

I should clarify that pip doesn't really create a true virtual environment (à la virtualenv/venv). It is indeed an isolated environment, but the isolation is achieved by the site/sys.path manipulations that are also the reason why cmake is broken here. Otherwise, it's really the same Python pip is running off.

Oh, this clarifies a lot for me. That is messy... I wonder why pip isn't using venv? Performance issue? Availability?

uv can make a real venv at almost no cost, so maybe they're doing that?

[ichard26]

Taking a brief look on the tracker for related reading nets these issues:

• Cannot install into user site directory with editable source. #7953
• Try switching to venv for isolation #10720
• pip fails to build project from backend-path when MetaPathFinder is present and installing from VCS url #11812

TL;DR, it's complicated. I don't plan on investigating this issue further.

[pfmoore]

IMO "use a proper venv" is the right solution here¹. Especially if it turns out that it's what uv does. But switching to venv is not trivial (as the links @ichard26 provided show) so we can't give any realistic timescale for when it will get done. A 3rd party PR might help, but I wouldn't necessarily expect an external contributor to be willing to navigate the complexities of our compatibility policies, so maintainer time could still be the bottleneck.

Footnotes

1. It might not actually solve all the problems we're seeing reports of, although if uv is using this approach, that would give us good evidence that it does. But at a minimum, it gives us the option of saying "blame venv" rather than having to patch up our own approach 🙂 ↩

[XuehaiPan]

I'm not sure we should allow users to run arbitrary external scripts in the build environment with shared sys.executable (even in a subprocess). If we want to support full isolation, I think creating a true virtual environment is the only solution.

The issue I'm facing is a narrower one that is limited to console scripts in venv/bin. I think there is an easy workaround to update the way we create the console scripts. What do you think?

UPDATE: This might not work because the console script could be installed by other installers rather than pip.

For example, inject the sys.path before importing the entrypoints. Since the console script is always run in a subprocess (it contains a shebang), it would not be a big problem to modify sys.path.

  #!${PROJECT}/venv/bin/python3.13
  # -*- coding: utf-8 -*-
  import re
  import sys
+ sys.path.insert(0, '${PROJECT}/venv/lib/python3.13/site-packages')
  from cmake import cmake
  if __name__ == '__main__':
      sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
      sys.exit(cmake())

Related source:

pip/src/pip/_vendor/distlib/scripts.py

Lines 41 to 49 in 93d43c9

	FIRST_LINE_RE = re.compile(b'^#!.*pythonw?[0-9.]*([ \t].*)?$')
	SCRIPT_TEMPLATE = r'''# -*- coding: utf-8 -*-
	import re
	import sys
	from %(module)s import %(import_name)s
	if __name__ == '__main__':
	sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
	sys.exit(%(func)s())
	'''

Note that we should handle the site-packages path carefully because there could be non-standard paths (e.g., dist-packages for Ubuntu system packages).

[notatallshaw]

UPDATE: This might not work because the console script could be installed by other installers rather than pip.

For example, inject the sys.path before importing the entrypoints. Since the console script is always run in a subprocess (it contains a shebang), it would not be a big problem to modify sys.path.

Yeah, whatever is changed here is only valid for things pip installed, and it would affect all things pip installed. I'd be worried this would break some other use case, and there would need to be convincing that it doesn't.

There's been significant discussion about making a much safer change to this entry-point script template (#13165) and it's currently undecided.

[XuehaiPan]

FYI, after some debugging effort, I find:

$ source venv/bin/activate
$ pip3 install cmake
$ pip3 install .  # fail (use cmake console script in venv/bin/camke)

$ source venv/bin/activate
$ pip3 install cmake
$ deactivate  # install without the venv activated
$ venv/bin/pip3 install .  # success (use system cmake binary)

---

Removing the PYTHONPATH environment variable before calling cmake also works under venv:

  # setup.py

  import os
  import shutil
  from pathlib import Path

  from setuptools import Extension, setup
  from setuptools.command.build_ext import build_ext


  HERE = Path(__file__).absolute().parent


  class CMakeExtension(Extension):
      def __init__(self, name, source_dir=".", target=None, **kwargs):
          super().__init__(name, sources=[], **kwargs)
          self.source_dir = Path(source_dir).absolute()
          self.target = target if target is not None else name.rpartition(".")[-1]

      @classmethod
      def cmake_executable(cls):
          cmake = os.getenv("CMAKE_EXECUTABLE", "")
          if not cmake:
              cmake = shutil.which("cmake")
          return cmake


  class cmake_build_ext(build_ext):
      def build_extension(self, ext):
          if not isinstance(ext, CMakeExtension):
              super().build_extension(ext)
              return
  
          cmake = ext.cmake_executable()
          if cmake is None:
              raise RuntimeError("Cannot find CMake executable.")

+         os.environ.pop('PYTHONPATH', None)
          self.spawn([cmake, "--version"])


  setup(
      name="cmake-venv-test",
      version="0.0.1",
      cmdclass={"build_ext": cmake_build_ext},
      ext_modules=[CMakeExtension("cmake_venv_test._C", source_dir=HERE)],
  )

$ source venv/bin/activate
$ pip3 install cmake
$ pip3 install .  # success

---

Here is a temporary workaround that works for me:

- self.spawn([cmake, '-S', str(ext.source_dir), '-B', str(build_temp), *cmake_args])
- if not self.dry_run:
-     self.spawn([cmake, '--build', str(build_temp), *build_args])
+ python_path = None
+ try:
+     # pip's build environment pseudo-isolation sets `PYTHONPATH` and may break console scripts
+     python_path = os.environ.pop('PYTHONPATH', None)  # unset `PYTHONPATH`
+     self.spawn([cmake, '-S', str(ext.source_dir), '-B', str(build_temp), *cmake_args])
+     if not self.dry_run:
+         self.spawn([cmake, '--build', str(build_temp), *build_args])
+ finally:
+     if python_path is not None:
+         os.environ['PYTHONPATH'] = python_path