Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parse dependencies from Python setup.cfg files #2281

Closed
wants to merge 1 commit into from

Conversation

akaihola
Copy link

@akaihola akaihola commented Jul 5, 2020

Fixes #2133

Also discussed here:
https://github.community/t/dependency-graph-does-not-support-setup-cfg-for-python/2576

This implementation modifies the parse_setup Python helper to use

distutils.core.run_setup(
    <path to unmodified setup.py file>,
    stop_after="init"
)

and the results are collected using the custom setup() function as before.

It also catches all exceptions. In case of errors or no collected dependencies, it reverts to the old method running a patched setup.py using exec() method.

I haven't yet added test cases for this. I'll need to install the Ruby environment and study how to run the test suite – pointers to documentation appreciated!

@@ -1,3 +1,4 @@
from distutils.core import run_setup

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Best to avoid using distutils, we're trying to actively deprecate it.

Copy link
Author

@akaihola akaihola Jul 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, so I take it that I should close this PR?

Is there an alternative best practice for getting the dependencies of a Python package which uses setuptools, no matter whether the dependencies are listed in setup.py or setup.cfg?

If there is, I'd like to add it as an answer to this Stack Overflow question which is one of the top search results when looking for solutions to read a Python package's dependencies.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This question is actually a bit difficult to answer. Assuming you actually just want to get the dependencies, for setuptools projects, the biggest issue you'll have is that many projects unfortunately don't know about (or haven't updated to use) environment markers, so it's distressingly common to see stuff like this:

requires = ["some_dependency"]
if sys.version_info[0] < 2:
     requires.append("some_py2_only_dependency")

E.g. from s3transfer. This means that depending on the machine where you run setup.py, you will get a different answer for the values of install_requires. The correct way to do this is to declare a fixed set of dependencies like so:

some_dependency
some_py2_only_dependency ; python_version < '2.7'

In which case even the conditional dependencies will be included in the wheel metadata.

Assuming that you are comfortable with accepting "whatever will be included in the wheel if I were to build it in whatever worker environment" (seems reasonable), then I would say that the best thing to do is to implement enough of a PEP 517/518 backend to execute prepare_metadata_for_build_wheel, and parse the dependency metadata from the .dist-info/METADATA file. The pep517 library should make this easy enough.

If the project doesn't have a pyproject.toml file, I recommend defaulting to use build-backend=setuptools.build_meta:__legacy__ and requires = ["setuptools >= 40.8.0", "wheel"], as we're doing in the python-build frontend. I think the majority of setuptools projects will work properly with these defaults. You can also fall back to using pip wheel to generate an actual wheel file and then extract the metadata from there, though that will consume more resources than necessary.

Using PEP 517 also has the advantage that it works automatically with any PEP 517 backend, not just setuptools. (We in the PyPA would also probably not be unhappy if you said, "We only support packages that work well with PEP 517", as a further spur to get people to actively adopt the new build system).

That said, I'm not exactly sure why you want this sort of generic information about a package's depdencies — my understanding of dependabot is that it is supposed to update your dependencies, no? So don't you need to know not only what they are, but also exactly where they are defined, so that you can make a PR to automatically update them?

Copy link
Author

@akaihola akaihola Jul 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my understanding of dependabot is that it is supposed to update your dependencies,

GitHub's Dependency Graph is implemented using dependabot. The way I arrived at this issue is that I wanted to simply see my project's dependencies in the GitHub UI and was surprised to see this message instead:

setup.py has no dependencies or is too large to display

So GitHub's Dependency Graph simply doesn't work for any Python packages which declare setup_requires in setup.cfg instead of setup.py.

I then found the tip to use distutils.core.run_setup(..., stop_after="init").install_requires and figured that dependabot could do that, and also as a side effect gain a possibly more robust way to extract dependencies from setup.py files than the current parse_setup() method which modifies setup.py and exec()s it in a patched Python environment. parse_setup() could still be used as a fallback for those cases where run_setup() fails.

It sounds like a great idea to make GitHub's Dependency Graph only work on modern well-formed Python packages. It should indeed encourage authors to adopt good practices in packaging. The current situation actually does the reverse – it discourages adopting e.g. a declarative setup.cfg and pyproject.toml and favors old-style packages with a traditional executable setup.py

Update: @jurre noted below that GitHub's Dependency Graph actually isn't based on dependabot at all. I must have misunderstood some information I've read about the Dependency Graph.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still eager to help make GitHub's Dependency Graph work for modern Python repositories. @pganssle could you advise what would be the most efficient and acceptable way for me to contribute?

Copy link

@graingert graingert Apr 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

import build.util
import importlib.metadata
import pathlib

def parse_from_setup_py(v: pathlib.Path) -> importlib.metadata.PackageMetadata:
    return build.util.project_wheel_metadata(v.parent)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, but I'm a complete Python noob, and I'm running into a bunch of missing import errors that I cannot seem to solve (on Python 3.6). So what I'd need it a complete example / standalone .py file that takes a directory / file as an argument and works out the box 🙄

Copy link

@graingert graingert Apr 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sschuberth what errors are you getting? Python 3.6 is EOL: you should try this on a recent version of python eg 3.9 or 3.10

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm stuck with Python 3.6, and I'm getting NameError for pathlib and importlib not being defined, AttributeError: module 'importlib' has no attribute 'metadata' after adding imports etc. I'll give https://stackoverflow.com/a/71276197/1127485 a try now.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For older versions, you have to install and use the backport importlib_metadata (notice the underscore)

@nschloe
Copy link

nschloe commented Apr 26, 2021

Can probably be closed now that #3423 has landed.

@jurre
Copy link
Member

jurre commented Jun 14, 2021

I think this has landed in #4323

@jurre jurre closed this Jun 14, 2021
akkornel added a commit to stanford-rc/mais-apis-python that referenced this pull request Jul 6, 2021
This enables Dependabot to run every Tuesday at 10 AM (Stanford time),
and notify us (by pull request) if a dependency has done an update.
It's then on us to check if the update mandates a bump in our minimum
required version for the dependency.

Kudos to dependabot/dependabot-core#2133,
dependabot/dependabot-core#2281, and
dependabot/dependabot-core#3423 for enabling
Dependabot support with `python.cfg` files!
@akaihola akaihola deleted the python-setup-cfg-support branch April 11, 2022 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for Python dependencies in setup.cfg