[Bug] Failing to install with pip because of recent change to setuptools

[smiledelf]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

pip install dbt-core (or dbt-databricks) will run into an import problem when it tries to install logbook.
ModuleNotFoundError: No module named 'setuptools.command.test'
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

After some investigation:

• dbt-core lists logbook==1.5 as a requirement
• logbook 1.5 tries to import setuptools.command.test but fails as it doesn't exist
• setuptools recently removed test command: commit, pypa/setuptools/issues/4519

Note: the latest logbook version (1.7) installs successfully so it looks like it's been fixed on their end.

Expected Behavior

python3 -m pip install dbt-core should install successfully.

Steps To Reproduce

1. With python virtual env: python3 -m venv venv
2. Activate: . venv/bin/activate
3. python3 -m pip install dbt-core
4. Get the error.

Relevant log output

`python3 -m pip install dbt-core`
Collecting dbt-core
  Downloading dbt_core-1.8.4-py3-none-any.whl.metadata (3.9 kB)
Collecting agate<1.10,>=1.7.0 (from dbt-core)
  Downloading agate-1.9.1-py2.py3-none-any.whl.metadata (3.2 kB)
Collecting Jinja2<4,>=3.1.3 (from dbt-core)
  Using cached jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
Collecting mashumaro<4.0,>=3.9 (from mashumaro[msgpack]<4.0,>=3.9->dbt-core)
  Using cached mashumaro-3.13.1-py3-none-any.whl.metadata (114 kB)
Collecting logbook<1.6,>=1.5 (from dbt-core)
  Using cached Logbook-1.5.3.tar.gz (85 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Traceback (most recent call last):

......

File "/tmp/pip-build-env-yxiain02/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-yxiain02/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 297, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-yxiain02/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 497, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-yxiain02/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 64, in <module>
      ModuleNotFoundError: No module named 'setuptools.command.test'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Environment

- OS: Ubuntu 24.04 LTS
- Python:3.12.3
- dbt: 1.8

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

databricks (trying to install dbt-databricks) but the same bug appears when installing dbt-core

[confidantduk]

workaround mentioned in this thread

[harukitake]

I think this problem will be solved if #10119 is released

[harukitake]

I have the same issue.

environment

• Ubuntu 22.04
• python 3.12.0
• dbt-core 1.7.17

[juan-abia]

Same issue here:

• python 3.11.6
• dbt-core 1.8.4
• setuptools 71.0.3

[toadjaune]

I think this problem will be solved if #10119 is released

I'm confused though, it has been merged since may, while the latest release is a week old

[stefanooldeman]

Perhaps something in poetry changed that invokes a different path during builds?
Update: it seems a newer version of pip is released over the weekend: https://pip.pypa.io/en/stable/news/

ChefBuildError
  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  Traceback (most recent call last):
    File "/opt/poetry/venv/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in <module>
      main()
    File "/opt/poetry/venv/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 357, in main
      json_out["return_val"] = hook(**hook_input["kwargs"])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/poetry/venv/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 134, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/tmp4tuj0f54/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=[])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/tmp4tuj0f54/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 297, in _get_build_requires
      self.run_setup()
    File "/tmp/tmp4tuj0f54/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 497, in run_setup
      super().run_setup(setup_script=setup_script)
    File "/tmp/tmp4tuj0f54/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 313, in run_setup
      exec(code, locals())
    File "<string>", line 64, in <module>
  ModuleNotFoundError: No module named 'setuptools.command.test'
  
  at /opt/poetry/venv/lib/python3.11/site-packages/poetry/installation/chef.py:164 in _prepare
      [160](****redacted****)│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      [162](****redacted****)│ 
      163│             if error is not None:
    → [164](****redacted****)│                 raise error from None
      [165](****redacted****)│ 
      166│             return path
      [167](****redacted****)│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with logbook (1.5.3) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "logbook (==1.5.3)"'.

Could we allow installation of logbook 1.6, it could have a fix already available:
https://github.com/getlogbook/logbook/blob/develop/CHANGES#L21

[junronglau]

Adding these before installation of dbt-core solved it for me

python -m pip install --upgrade pip-tools pip wheel
python -m pip install setuptools==71.1.0 poetry==1.8.3

[mkielar]

I think this problem will be solved if #10119 is released

I'm confused though, it has been merged since may, while the latest release is a week old

They haven't removed the dependency, though, @toadjaune. It's still there in 1.8.4.
https://github.com/dbt-labs/dbt-core/blob/v1.8.4/core/setup.py#L55-L58

[rbenomar]

I think this problem will be solved if #10119 is released

I'm confused though, it has been merged since may, while the latest release is a week old

They haven't removed the dependency, though, @toadjaune. It's still there in 1.8.4. https://github.com/dbt-labs/dbt-core/blob/v1.8.4/core/setup.py#L55-L58

there's a dependabot pull request from a year ago:
#8254

[D3nn3]

fyi: the "faulty" setuptools release was yanked and everything should work as expected again.

pypa/setuptools#4519 (comment)
https://pypi.org/project/setuptools/#history

[emmyoop]

As @D3nn3 said, setuptools has yanked their release and dbt-core can be pip installed again.

To clarify some confusion in the comments, the logbook dependency in dbt-core will be removed when we release version 1.9.

[CincyBC]

The setuptools package released 72.1.0 about an hour ago, which we were able to successfully use to install the Logbook dependency.