Add support for Python dependencies in setup.cfg

[althonos]

Hi !

Ever since setuptools started supporting setup configuration files, I've been using them instead of hardcoding configuration values in the setup.py script. However, it does not seem like these files are supported by Dependabot right now.

Adding support would be fairly simple: check for setup.cfg, parse it as an INI configuration file, and check the value in setup_requires, tests_require, and install_requires just like a requirements.txt file.

I've been using Dependabot extensively for my Rust projects but this is stopping me from extending that to my Python projects as well. I'd love to see setup.cfg support implemented!

[greysteil]

Oh good point. We have a kind of partial support for them, where we can handle libraries that use them alongside a requirement.txt, but don't parse them or update them. We should.

[greysteil]

@althonos can you link to a repo that uses this setup? Would help a lot with adding support.

[althonos]

@greysteil : sure ! althonos/InstaLooter has that kind of structure, with every dependency listed in setup.cfg.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within seven days. Thank you for your contributions.

[althonos]

@greysteil : bumping ? 😄

[greysteil]

I'm not on team Dependabot anymore (I now work on security more broadly at GitHub). @hmarr, @feelepxyz and @jeffmcaffer are your go-tos now :-)

[ivelin]

@feelepxyz and @jeffmcaffer , could you please share an update on this issue.
setup.cfg is becoming popular in the python packaging community. Here is another example project that uses setup.cfg alongside setup.py for dependency management.

Thank you!

[feelepxyz]

@ivelin 👋 no progress on this yet. We're currently running a very small team maintaining dependabot alongside integrating it within GitHub so have made little progress on new features. We've got a few python improvements lined up once we get more people on the team.

[ivelin]

Thank you for the candid response, @feelepxyz . I understand and will keep an eye on the issue for updates.

[ivelin]

BTW, not sure if its an option, but if the Python setup parsing code in dependabot is open source, maybe you can provide a couple of hints and pointers. There may be contributors in the Python community with spare cycles to help out.

[feelepxyz]

@ivelin sorry for the slow update! Would love help adding support to dependabot-core. As a starting point, you probably want to hook in here and branch out to a new updater for setup.cfg: https://github.com/dependabot/dependabot-core/blob/master/python/lib/dependabot/python/file_updater.rb#L27-L43

You can use the requirements updater as a starting point: https://github.com/dependabot/dependabot-core/blob/master/python/lib/dependabot/python/file_updater/requirement_file_updater.rb

[mauvilsa]

Hi. Is there any update on plans for working on this? I see that the links that @ivelin posted as a starter for anyone willing to help are now broken. The new ones are:

• https://github.com/dependabot/dependabot-core/blob/main/python/lib/dependabot/python/file_updater.rb#L27-L43
• https://github.com/dependabot/dependabot-core/blob/main/python/lib/dependabot/python/file_updater/requirement_file_updater.rb

I also add a couple of example projects that use setup.cfg for requirements which include both install_requires and extras_require.

• https://github.com/omni-us/jsonargparse/blob/master/setup.cfg
• https://github.com/omni-us/narchi/blob/master/setup.cfg

I was considering helping out. But I don't know ruby so looking at the starting point it did not seem too easy for me to do. I also found https://github.com/datafolklabs/ruby-parseconfig which maybe is able to parse the setup.cfg files.

[akaihola]

This feature is also discussed here:
https://github.community/t/dependency-graph-does-not-support-setup-cfg-for-python/2576

PR #2281 modifies the parse_setup Python helper to use

distutils.core.run_setup(
    <path to unmodified setup.py file>,
    stop_after="init"
)

and the results are collected using the custom setup() function as before.

It also catches all exceptions. In case of errors or no collected dependencies, it reverts to the old method running a patched setup.py using exec() method.

I haven't yet added test cases for this. I'll need to install the Ruby environment and study how to run the test suite – pointers to documentation appreciated!

[graingert]

it would be better to use pep517 https://www.python.org/dev/peps/pep-0517/#prepare-metadata-for-build-wheel

[graingert]

here's a demo using the pep517.meta helper:

>>> import pep517.meta
>>> import pprint
>>> pprint.pprint(pep517.meta.load("path/to/twisted").requires)
['zope.interface (>=4.4.2)',
 'constantly (>=15.1)',
 'incremental (>=16.10.1)',
 'Automat (>=0.8.0)',
 'hyperlink (>=17.1.1)',
 'attrs (>=19.2.0)',
 'twisted-iocpsupport (~=1.0.0) ; platform_system == "Windows"',
...

[akaihola]

@graingert, have you taken a look at the other ideas thrown around in #2281? Could you add this one in that discussion?

[vikahl]

It would certainly be nice to have support for setup.cfg-files.

What would be the next steps to get this feature? There are different ideas suggested above, and I think #2281 is the furthest developed one?

[honnix]

Trying this again in #3423 (wip). Not entirely sure the approach would work.

[honnix]

I think this issue can be closed now.

[althonos]

Hooray! Thanks @honnix !

[cmichelenstrofer]

Dependencies are still not being shown in my Dependencies Graph in GitHub.

[michael-robbins]

I still have repos that have dependencies specified in setup.cfg where dependabot is not detecting and raising PRs. Is there any documentation for this?

[Rogdham]

If you are looking for dependencies being listed in the dependencies graph in GitHub, this issue seems to be irrelevant: unfortunately they are not using dependabot for that.

It's actually implemented separately. We hope to merge whatever is possible from these two implementations at some point, but we currently have no concrete plans to do that. Unfortunately any changes made to dependabot won't make it back into the dependency graph.

Related:

• Topic on GitHub community
• GitHub docs listing supported ecosystems
• Source of the citation above by @jurre (you need to expend some messages)

[kojiromike]

I'm a little confused about this issue. The subject seems to imply that dependabot does not support dependencies in setup.cfg, but I use it regularly with install_requires and extras as well, in setup.cfg. So I must be missing some nuance. Could someone update the subject and description of this issue to describe the current state of affairs if this should still be open?

[michael-robbins]

Having your dependencies defined in setup.cfg and having the pip package-ecosystem defined in dependabot.yml results in Dependabot not understanding/seeing any dependencies/ignoring setup.cfg.

It is still recognising the other github-actions configuration, so Dependabot is monitoring the repo, it's just ignoring setup.cfg and the dependencies identified.

You can see by navigating to the Dependency graph section it completely doesn't see setup.cfg: https://github.com/emojirades/emojirades/network/dependencies

This is the problem.

[akaihola]

It's been mentioned before that apparently GitHub's Dependency graph doesn't use Dependabot.

[mauvilsa]

The following is specifically about github's dependency graph and setup.cfg support community/community#6456. Would be good if more people comment there.

[DanielNoord]

I think this issue can be closed now.

This is still relevant. Somebody should probably close this issue 😄

[jurre]

Thanks @DanielNoord, I'll indeed close this out, please refer to https://github.com/orgs/github-community/discussions/6456 for discussion on support in the Dependency Graph (which as mentioned indeed currently does not share any implementation details with Dependabot)

[ssbarnea]

@DanielNoord While setup.cfg is still no1 place (popularity) for dependencies, this is going away in the next couple of years. We already have a PEP describing dependencies in pyproject.toml and if I understood correctly, pip already implemented initial support for it.

The really bad part is the overall status of dependabot and huge lagging behind everywhere. For example they do not support Yarn newer lock file formats even more than two years after these were introduced.

I seen ourselves slowly moving away from dependabot for these reasons. It still works good for updating github actions, so it is not totally useless. For python, we do use it in a bit different approach, where is still usable. We use pip-compile (from pip-tools) to compile constraints files, which we name requirements.txt. This means that dependabot will recognize them and update them, even if they are constraints, mostly because constraints files are always valid requirements files (the opposite not being true).

[DanielNoord]

Agreed that lagging behind of dependabot can become problematic. I do think it is still useful and probably one of the better tools out there. See also #5324, they are actively working on fixing some of the most prominent issues.

I think one of the issues is that dependabot is written in Ruby. Personally I have no experience in Ruby so other than providing context and test cases I can't do much to increase depenedabots support for newer Python patterns.

I do think we as Python community/ecosystem can help by providing clear test cases or repositories. For example in the PR mentioned above one of the issues is that we didn't find the correct test/pattern immediately. I can imagine that maintainers of dependabot don't know all the specifics for all the ecosystems they support, which is something we can help with 😄

[merwok]

We already have a PEP describing dependencies in pyproject.toml and if I understood correctly, pip already implemented initial support for it.

It is build tools (flit-core, hatchling, setuptools, etc) that have gained or are gaining support to find the project metadata and dependencies in pyproject.toml instead of each defining its own config file.
pip is an installer, it looks at metadata from the other side: PKG-INFO or METADATA file in sdists and wheels.

(pip also looks at pyproject.toml to know what build tool to install and call when building from source)