[BUG] SQLAlchemy editable install failure references setuptools PEP 660

[hab6]

setuptools version

63.2.0

Python version

3.10.7

OS

MSYS_NT-10.0-19042 3.3.6-341.x86_64

Additional environment information

pip v22.3, SQLAlchemy v1.4.42

Description

Attempt to build an editable install of project SQLAlchemy encounters an error due to a missing C++ compiler, which we expect, but then a setuptools problem is reported and references PEP 660.

References to similiar issues:

#3606 [Question] Editable install fails after introducing pyproject.toml
#3607 [BUG] If you are seeing this error, please report to
#3597 Struggling with fairseq installation

Expected behavior

The expectation is no error and a successfully built editable install, which does occur if using the suggested workaround with environment variable SETUPTOOLS_ENABLE_FEATURES="legacy-editable".

(py3_sa) C:\Users\jdoe\Data\vm\py3_sa\test\sqlalchemy>echo %SETUPTOOLS_ENABLE_FEATURES%
"legacy-editable"

(py3_sa) C:\Users\jdoe\Data\vm\py3_sa\test\sqlalchemy>python -m pip install -e .
Obtaining file:///C:/Users/jdoe/Data/vm/py3_sa/test/sqlalchemy
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting greenlet!=0.4.17
  Using cached greenlet-1.1.3.post0-cp310-cp310-win_amd64.whl (101 kB)
Installing collected packages: greenlet, SQLAlchemy
  Running setup.py develop for SQLAlchemy
Successfully installed SQLAlchemy-1.4.41.dev0 greenlet-1.1.3.post0

How to Reproduce

0. Ensure tools are installed (pip, python, git, setuptools, etc.)
1. git clone -b rel_1_4_41 https://github.com/sqlalchemy/sqlalchemy.git
2. cd sqlalchemy
3. python -m pip install -e .

Output

(py3_sa) λ git --version
git version 2.38.0.windows.1
(py3_sa) λ pip --version
pip 22.3 from C:\Users\jdoe\AppData\Local\Programs\Python\Python310\lib\site-packages\pip (python 3.10)
(py3_sa) λ python --version
Python 3.10.7

(py3_sa) λ git clone -b rel_1_4_41 https://github.com/sqlalchemy/sqlalchemy.git
Cloning into 'sqlalchemy'...
remote: Enumerating objects: 207671, done.
remote: Counting objects: 100% (2550/2550), done.
remote: Compressing objects: 100% (884/884), done.
remote: Total 207671 (delta 1577), reused 2298 (delta 1420), pack-reused 205121
Receiving objects: 100% (207671/207671), 88.69 MiB | 10.60 MiB/s, done.
Resolving deltas: 100% (156035/156035), done.

(py3_sa) λ cd sqlalchemy

(py3_sa) C:\Users\jdoe\Data\vm\py3_sa\test\sqlalchemy>python -m pip install -e .
Obtaining file:///C:/Users/jdoe/Data/vm/py3_sa/test/sqlalchemy
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting greenlet!=0.4.17
  Using cached greenlet-1.1.3.post0-cp310-cp310-win_amd64.whl (101 kB)
Building wheels for collected packages: SQLAlchemy
  Building editable for SQLAlchemy (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building editable for SQLAlchemy (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [77 lines of output]
      C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\config\setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
        warnings.warn(msg, warning_class)
      running editable_wheel
      creating C:\Users\jdoe\AppData\Local\Temp\pip-wheel-pcpwy2ic\tmpx6tg15ho\SQLAlchemy.egg-info
      writing C:\Users\jdoe\AppData\Local\Temp\pip-wheel-pcpwy2ic\tmpx6tg15ho\SQLAlchemy.egg-info\PKG-INFO
      writing dependency_links to C:\Users\jdoe\AppData\Local\Temp\pip-wheel-pcpwy2ic\tmpx6tg15ho\SQLAlchemy.egg-info\dependency_links.txt
      writing requirements to C:\Users\jdoe\AppData\Local\Temp\pip-wheel-pcpwy2ic\tmpx6tg15ho\SQLAlchemy.egg-info\requires.txt
      writing top-level names to C:\Users\jdoe\AppData\Local\Temp\pip-wheel-pcpwy2ic\tmpx6tg15ho\SQLAlchemy.egg-info\top_level.txt
      writing manifest file 'C:\Users\jdoe\AppData\Local\Temp\pip-wheel-pcpwy2ic\tmpx6tg15ho\SQLAlchemy.egg-info\SOURCES.txt'
      reading manifest file 'C:\Users\jdoe\AppData\Local\Temp\pip-wheel-pcpwy2ic\tmpx6tg15ho\SQLAlchemy.egg-info\SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching '*.html' under directory 'doc'
      warning: no files found matching '*.css' under directory 'doc'
      warning: no files found matching '*.js' under directory 'doc'
      no previously-included directories found matching 'doc\build\output'
      adding license file 'LICENSE'
      writing manifest file 'C:\Users\jdoe\AppData\Local\Temp\pip-wheel-pcpwy2ic\tmpx6tg15ho\SQLAlchemy.egg-info\SOURCES.txt'
      creating 'C:\Users\jdoe\AppData\Local\Temp\pip-wheel-pcpwy2ic\tmpx6tg15ho\SQLAlchemy-1.4.41.dev0.dist-info'
      C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\wheel\bdist_wheel.py:80: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
        if get_flag('Py_DEBUG',
      creating C:\Users\jdoe\AppData\Local\Temp\pip-wheel-pcpwy2ic\tmpx6tg15ho\SQLAlchemy-1.4.41.dev0.dist-info\WHEEL
      running build_py
      running build_ext
      building 'sqlalchemy.cprocessors' extension
      Traceback (most recent call last):
        File "<string>", line 74, in build_extension
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\command\build_ext.py", line 246, in build_extension
          _build_ext.build_extension(self, ext)
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 547, in build_extension
          objects = self.compiler.compile(
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\_distutils\_msvccompiler.py", line 344, in compile
          self.initialize()
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\_distutils\_msvccompiler.py", line 253, in initialize
          vc_env = _get_vc_env(plat_spec)
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\msvc.py", line 214, in msvc14_get_vc_env
          return _msvc14_get_vc_env(plat_spec)
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\msvc.py", line 168, in _msvc14_get_vc_env
          raise distutils.errors.DistutilsPlatformError(
      distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 140, in run
          self._create_wheel_file(bdist_wheel)
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 330, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 261, in _run_build_commands
          self._run_build_subcommands()
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 288, in _run_build_subcommands
          self.run_command(name)
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 319, in run_command
          self.distribution.run_command(command)
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\dist.py", line 1217, in run_command
          super().run_command(command)
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 987, in run_command
          cmd_obj.run()
        File "<string>", line 68, in run
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\command\build_ext.py", line 84, in run
          _build_ext.run(self)
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 346, in run
          self.build_extensions()
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 466, in build_extensions
          self._build_extensions_serial()
        File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 492, in _build_extensions_serial
          self.build_extension(ext)
        File "<string>", line 76, in build_extension
      BuildFailed
      error: Support for editable installs via PEP 660 was recently introduced
      in `setuptools`. If you are seeing this error, please report to:

      https://github.com/pypa/setuptools/issues

      Meanwhile you can try the legacy behavior by setting an
      environment variable and trying to install again:

      SETUPTOOLS_ENABLE_FEATURES="legacy-editable"
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building editable for SQLAlchemy
Failed to build SQLAlchemy
ERROR: Could not build wheels for SQLAlchemy, which is required to install pyproject.toml-based projects

[clach04]

It looks like the C-compiler check is always happening, even though it's not actually needed.

  Traceback (most recent call last):
    File "<string>", line 74, in build_extension
    File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\command\build_ext.py", line 246, in build_extension
      _build_ext.build_extension(self, ext)
    File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 547, in build_extension
      objects = self.compiler.compile(
    File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\_distutils\_msvccompiler.py", line 344, in compile
      self.initialize()
    File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\_distutils\_msvccompiler.py", line 253, in initialize
      vc_env = _get_vc_env(plat_spec)
    File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\msvc.py", line 214, in msvc14_get_vc_env
      return _msvc14_get_vc_env(plat_spec)
    File "C:\Users\jdoe\AppData\Local\Temp\pip-build-env-w9ttab3p\overlay\Lib\site-packages\setuptools\msvc.py", line 168, in _msvc14_get_vc_env
      raise distutils.errors.DistutilsPlatformError(
  distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

[abravalheri]

Hi @hab6 thank you for reporting this.

Am I correct to assume that no error happens if you run python -m pip install . (i.e. without the -e flag) in a fresh clone of the repository?

[hab6]

Thanks for the quick response @abravalheri. Correct, without the -e flag, the install runs successfully.

See below output

1. Fresh clone of repository
2. python -m pip install . # SUCCESS
3. python -m pip install -e . #FAIL

C:\Users\jdoe\Data\pvm\py3_sa\test\a
(py3_sa) λ git clone -b rel_1_4_41 https://github.com/sqlalchemy/sqlalchemy.git
Cloning into 'sqlalchemy'...
remote: Enumerating objects: 207705, done.
remote: Counting objects: 100% (2584/2584), done.
remote: Compressing objects: 100% (904/904), done.
remote: Total 207705 (delta 1596), reused 2321 (delta 1429), pack-reused 205121
Receiving objects: 100% (207705/207705), 88.55 MiB | 11.54 MiB/s, done.
Resolving deltas: 100% (156075/156075), done.

C:\Users\jdoe\Data\pvm\py3_sa\test\a
(py3_sa) λ cd sqlalchemy

C:\Users\jdoe\Data\pvm\py3_sa\test\a\sqlalchemy(HEAD detached at ac9a74a)
(py3_sa) λ python -m pip install .
Processing c:\users\jdoe\data\pvm\py3_sa\test\a\sqlalchemy
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: greenlet!=0.4.17 in c:\users\jdoe\appdata\local\programs\python\python310\lib\site-packages (from SQLAlchemy==1.4.41.dev0) (1.1.3.post0)
Building wheels for collected packages: SQLAlchemy
  Building wheel for SQLAlchemy (pyproject.toml) ... done
  Created wheel for SQLAlchemy: filename=SQLAlchemy-1.4.41.dev0-cp310-cp310-win_amd64.whl size=1544962 sha256=8de29383f1903c51d28e884ce87bb694795d9d2fbcaec3d6fa0fd88a094307a7
  Stored in directory: C:\Users\jdoe\AppData\Local\Temp\pip-ephem-wheel-cache-4fktlqhj\wheels\61\de\1c\e81b97aff90b3d50e9a1734a88ab7deda8acc14a4a30b18105
Successfully built SQLAlchemy
Installing collected packages: SQLAlchemy
  Attempting uninstall: SQLAlchemy
    Found existing installation: SQLAlchemy 1.4.41.dev0
    Uninstalling SQLAlchemy-1.4.41.dev0:
      Successfully uninstalled SQLAlchemy-1.4.41.dev0
Successfully installed SQLAlchemy-1.4.41.dev0

C:\Users\jdoe\Data\pvm\py3_sa\test\a\sqlalchemy(HEAD detached at ac9a74a)
(py3_sa) λ python -m pip install -e .
Obtaining file:///C:/Users/jdoe/Data/pvm/py3_sa/test/a/sqlalchemy
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: greenlet!=0.4.17 in c:\users\jdoe\appdata\local\programs\python\python310\lib\site-packages (from SQLAlchemy==1.4.41.dev0) (1.1.3.post0)
Building wheels for collected packages: SQLAlchemy
  Building editable for SQLAlchemy (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building editable for SQLAlchemy (pyproject.toml) did not run successfully......
  │ exit code: 1                                                                    
.....
  BuildFailed                                                             
      error: Support for editable installs via PEP 660 was recently introduced
      in `setuptools`. If you are seeing this error, please report to:        
                                                                              
      https://github.com/pypa/setuptools/issues                               
.....

[abravalheri]

Thank you very much @hab6, I will have a look on this problem. It sounds a bit complicated, specially to get an available Windows machine with the same characteristics, so it may take some time, sorry for that.

It looks like the C-compiler check is always happening, even though it's not actually needed.

@clach04, I am trying to understand this statement. I had a look on SQLAlchemy and it seems that they use Cython, so there is some compilation going on, right? Do you have any extra information why the compiler checks would not be needed?

[hab6]

@abravalheri Cython may not be relevant to this issue since there are no Cython source (*.pyx) files in SQLAlchemy v1.4.41 (whereas SQLAlchemy v2.0 does contain *.pyx files).

SQLAlchemy v1.4.41 clone

λ grep Version lib\SQLAlchemy.egg-info\PKG-INFO | grep dev0
Version: 1.4.41.dev0

(py3C) λ dir /s *.pyx
 Volume in drive C is Windows
 Volume Serial Number is E036-DF48
File Not Found

SQLAlchemy v2.0.0 clone

λ grep Version lib\SQLAlchemy.egg-info\PKG-INFO | grep dev0
Version: 2.0.0b3.dev0

λ dir /s *.pyx
 Volume in drive C is Windows
 Volume Serial Number is E036-DF48

 Directory of C:\Users\jdoe\Data\pvm\py3_sa\test2\sqlalchemy\lib\sqlalchemy\cyextension

10/21/2022  04:06 PM            12,268 collections.pyx
10/21/2022  04:06 PM             3,295 immutabledict.pyx
10/21/2022  04:06 PM             1,607 processors.pyx
10/21/2022  04:06 PM             2,768 resultproxy.pyx
10/21/2022  04:06 PM             2,374 util.pyx
               5 File(s)         22,312 bytes

[clach04]

Adding to the above comment, the (in development, i.e. no stable) 2.x in main does have the option to use Cython but it's currently not-mandatory.

• cython is not listed as a required dependency in the setup scripts
• https://github.com/sqlalchemy/sqlalchemy/blob/main/setup.py#L13 makes it optional and only uses if available

so cython is a red-herring (at least with the current situation in October 2022).

Both pip install . and set SETUPTOOLS_ENABLE_FEATURES="legacy-editable" ; pip install -e . don't trigger the compiler check (and failure). So at least SA can be installed while this is figured out :-).

Kudos to whom ever added the legacy message, saved a bunch of time and bother figuring out a workaround! 👍 And thanks to @abravalheri for jumping on this so quickly!

[abravalheri]

Thank you very much for the extra information guys. For the sake of transparency (and to manage expectations) I just would like to let you know that I will be away from the computer for the majority of this week, so I will not be able to work on this specifically.

[abravalheri]

cython is not listed as a required dependency in the setup scripts\nhttps://github.com/sqlalchemy/sqlalchemy/blob/main/setup.py#L13 makes it optional and only uses if available

Thank you very much @clach04 , I had a quick look in the repo from my phone and I can see that Cython is listed in pyproject.toml and will always be installed in CPython:

https://github.com/sqlalchemy/sqlalchemy/blob/main/pyproject.toml#L6

If I am not mistaking something, this means setuptools will always try to use Cython and build binary extensions, unless the user is using an alternative Python implementation (e.g. PyPy, IronPython, Jython, etc...)

In practice, although Cython is "not mandatory", the build configuration is kind of forcing it to be used, right?

It's a very weird (and this is the part I will try to investigate/fix once I am back to the computer) that the editable install differs from the regular install.

[clach04]

cython is not listed as a required dependency in the setup scripts\nhttps://github.com/sqlalchemy/sqlalchemy/blob/main/setup.py#L13 makes it optional and only uses if available

Thank you very much @clach04 , I had a quick look in the repo from my phone and I can see that Cython is listed in pyproject.toml and will always be installed in CPython:

https://github.com/sqlalchemy/sqlalchemy/blob/main/pyproject.toml#L6

If I am not mistaking something, this means setuptools will always try to use Cython and build binary extensions, unless the user is using an alternative Python implementation (e.g. PyPy, IronPython, Jython, etc...)

In practice, although Cython is "not mandatory", the build configuration is kind of forcing it to be used, right?

It's a very weird (and this is the part I will try to investigate/fix once I am back to the computer) that the editable install differs from the regular install.

Ahh, didn't spot that in the main branch. Thanks! I'll remember to check that in future.

I can confirm that for @hab6's reported case, Cython is not used nor listed as a dependency the https://github.com/sqlalchemy/sqlalchemy/blob/rel_1_4/pyproject.toml in the rel_1_4_41/rel_1_4 branch(es) do not list nor use Cython so I do not believe the build system is forcing the use of Cython with the reported test case. But for a new test case of the main (default) branch I suspect you are correct, that might then be a different problem to investigate (that might be combination of a setuptools and sqlalchemy issue).

[abravalheri]

I can confirm that for @hab6's reported case, Cython is not used nor listed as a dependency the https://github.com/sqlalchemy/sqlalchemy/blob/rel_1_4/pyproject.toml in the rel_1_4_41/rel_1_4 branch(es) do not list nor use Cython so I do not believe the build system is forcing the use of Cython with the reported test case.

Thank you very much for the clarification @clach04.

I think I know what is happening now, or at least I know why setuptools is running the compiler validation.

In the rel_1_4 tag, the setup.py will always try to build the extension (there is no conditional for Cython). This will trigger the compiler check.

I noticed that SQALachemy's setup.py script in rel_1_4 is confident that a BuildFailed exception is the only kind of exception that is worth trying to catch and rebuild the wheel without the extensions.

When a developer tries to install the package in editable mode, setuptools will raise a different setuptools.errors.InternalError object, which is not handled by the setup.py script. This is why the build fails and the try-except for building without the extensions does not work.

If I am not wrong this part of the issue might be a duplicate of #3572. My suggestion there was to use the optional=True keyword for the Extension in the setup script instead of doing a try-except.

---

@hab6 if the missing compiler is a well know limitation for this build on windows, I think we can stop the investigation here. Effectively the issue is the same as the one discussed in #3572 and the solutions described there would apply.

If the compiler tools are installed we can continue the investigation and try to understand what is going wrong... Could you please let me know which circumstance we are working with here?

[hab6]

@abravalheri Greatly appreciate your informative help! My environment does not have the Cython tools installed. In looking at #3572, I think it stems from (and was opened 2 days after) SQLAlchemy issue #8458. So the SQLAlchemy team is aware of this issue and appears to have delivered a fix to address the problem in rel_2_0_0b1.

Based on this information, I think we can close this issue. @clach04 Do you concur?

[abravalheri]

Thank you very much @hab6, @clach04.

I will close this issue in favour of the solution discussed in #3572.
Please let me know if there is a different problem that we need to investigate.