[CT-2027] [Feature] Standardize setup.py across adapter repos

[dbeatty10]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Loving the changes that @mikealfare made to setup.py in Redshift.

It seems like we should standardize and make similar changes in the other adapters!

• dbt-bigquery/setup.py
• dbt-snowflake/setup.py
• dbt-spark/setup.py

I'm not sure if they would apply to Postgres or not 🤷 -- someone would need to assess:

• postgres/setup.py

Bonus suggestion:

• put the definition of setup.py here and then just use that to stamp out instances for each adapter

Describe alternatives you've considered

We could allow the structure and contents of setup.py to drift. But I don't think it's necessary for them to drift. After a detailed assessment, nearly everything I saw was undifferentiated across adapters.

Who will this benefit?

One benefit:

• My assumption is that some maintainers of the adapters listed here largely take their setup.py from an adapter maintained by dbt Labs, and it would be nice for them to get the "latest and greatest" version.

Another benefit:

• Folks have run into problems that standardized setup.py could have prevented:
  • Example: v1.0.3 pinned to dbt-core==1.0.3 starburstdata/dbt-trino#221

Are you interested in contributing this feature?

Happy to help however -- pretty easy to open PRs for these

Anything else?

See below for a templated version of setup.py (that may itself be stale already 🤷). It would be very easy to convert this for usage by Copier, Cruft, Cookiecutter, Cookieninja, etc.

Cookiecutter template for

setup.py

#!/usr/bin/env python
import sys

if sys.version_info < (3, 7):
    print("Error: dbt does not support this version of Python.")
    print("Please upgrade to Python 3.7 or higher.")
    sys.exit(1)


try:
    from setuptools import find_namespace_packages
except ImportError:
    print("Error: dbt requires setuptools v40.1.0 or higher.")
    print('Please upgrade setuptools with "pip install --upgrade setuptools" and try again')
    sys.exit(1)


from pathlib import Path
from setuptools import setup


# pull the long description from the README
README = Path(__file__).parent / "README.md"


# used for this adapter's version and in determining the compatible dbt-core version
VERSION = Path(__file__).parent / "dbt/adapters/{{ cookiecutter.directory_name }}/__version__.py"


def _plugin_version() -> str:
    """
    Pull the package version from the main package version file
    """
    attributes = {}
    exec(VERSION.read_text(), attributes)
    return attributes["version"]


def _core_patch(plugin_patch: str):
    """
    Determines the compatible dbt-core patch given this plugin's patch

    Args:
        plugin_patch: the version patch of this plugin
    """
    pre_release_phase = "".join([i for i in plugin_patch if not i.isdigit()])
    if pre_release_phase:
        if pre_release_phase not in ["a", "b", "rc"]:
            raise ValueError(f"Invalid prerelease patch: {plugin_patch}")
        return f"0{pre_release_phase}1"
    return "0"


# require a compatible minor version (~=) and prerelease if this is a prerelease
def _core_version(plugin_version: str = _plugin_version()) -> str:
    """
    Determine the compatible dbt-core version given this plugin's version.

    We assume that the plugin must agree with `dbt-core` down to the minor version.

    Args:
        plugin_version: the version of this plugin, this is an argument in case we ever want to unit test this
    """
    try:
        major, minor, plugin_patch = plugin_version.split(".")
    except ValueError:
        raise ValueError(f"Invalid version: {plugin_version}")

    return f"{major}.{minor}.{_core_patch(plugin_patch)}"


setup(
    name="{{ cookiecutter.project_name }}",
    version=_plugin_version(),
    description="The {{ cookiecutter.platform_short_name }} adapter plugin for dbt",
    long_description=README.read_text(),
    long_description_content_type="text/markdown",
    author="dbt Labs",
    author_email="info@dbtlabs.com",
    url="[https://github.com/dbt-labs/{{](https://github.com/dbt-labs/%7B%7B) cookiecutter.project_name }}",
    packages=find_namespace_packages(include=["dbt", "dbt.*"]),
    include_package_data=True,
    install_requires=[
        f"dbt-core~={_core_version()}",
    ],
    zip_safe=False,
    classifiers=[
        "Development Status :: 5 - Production/Stable",
        "License :: OSI Approved :: Apache Software License",
        "Operating System :: Microsoft :: Windows",
        "Operating System :: MacOS :: MacOS X",
        "Operating System :: POSIX :: Linux",
        "Programming Language :: Python :: 3.7",
        "Programming Language :: Python :: 3.8",
        "Programming Language :: Python :: 3.9",
        "Programming Language :: Python :: 3.10",
        "Programming Language :: Python :: 3.11",
    ],
    python_requires=">=3.7",
)

[emmyoop]

Part of these updates should include visiting the conditionals in setup.py per dbt-labs/dbt-snowflake#633

[DustinMoriarty]

If setup.py is being overhauled, why not just switch to pyproject.toml and maybe also switch to python poetry at the same time?

[dbeatty10]

@DustinMoriarty we're definitely interested in switching to pyproject.toml 🤩 We have some light discussion in dbt-labs/dbt-core#5696 and dbt-labs/dbt-core#4674 related to that switch. I think this will happen, and it's just a matter of when.

We'd consider Poetry separately. We last discussed this in dbt-labs/dbt-core#4446, which has since gone stale and subsequently closed. We'd invite you to comment on that closed issue and we can re-open it for further discussion. I just added a comment here.

[DustinMoriarty]

@dbeatty10 : I would definitely recommend poetry for any project, especially one that is refactoring it's build tools. I would be happy to help with the migration if you want. I have migrated quite a few projects to poetry. It is not nearly as big of a change as you would think. It is not a breaking change because the build output meets the same PEP standards. The project just becomes easier to manage.

[benc-db]

We (Databricks) have a stale PR in our repo for doing exactly this. It's mostly never gotten merged because I waffled about if we're ok with more fragmentation for people who might work across the ecosystem.

[benc-db]

In other words, would strongly appreciate the move to modern build tools. Poetry's value is maybe more debatable, but I'd love to consolidate all our python config into a pyproject.toml.

[dbeatty10]

We migrated to hatch in #17 for this repo.

dbt-postgres has an example pyproject.toml.

When we do the same for the other adapters maintained by dbt Labs, then this issue will be resolved.