Python updater appears to produce incorrect behaviour when multiple requirements.in, requirements.txt and setup.py exist

[JimNero009]

Hello,

I have observed the following behaviour from dependabot in a repo setup like this:

requirements/
  requirements-dev-dbconnect.in
  requirements-dev-unit.in
  requirements-dev.in
  requirements.in

requirements-dev-dbconnect.txt
requirements-dev-unit.txt
requirements-dev.txt
requirements.txt
setup.py (empty w.r.t. install_requires)

I appear to find that, when dependabot scans this repo and detects it as a pip-compile setup, it essentially runs incorrect commands when it tries to bump and recompile. As a human, you can reason what the mapping between the requirements.in files and requirements.txt files should be, but dependabot lacks this context and appears to find all the relevant files but then apply them incorrectly (for example, it will find the setup.py and the empty install_requires list within and generate an empty requirements.txt). I tried to debug this as best as I could but my lack of Ruby experience meant I found it difficult to follow the entire execution and reason on every decision it makes, but from what I could understand it would seem that dependencies are parsed out of all the correct places, but then updated and the lockfiles regenerated incorrectly.

I would be interested in knowing how the python bumping code is supposed to deal with situations where there are multiple requirements.in and requirements.txt files. Is this just not supported? Is there something I can do to explain to dependabot which files map to which? Is there a way I can rewrite things to avoid the need for multiple files entirely?

For further context (I cannot link the project since it is internal to a enterprise instance), the project in question is set up like this so that they can independently generate different, non-conflicting requirements files so the user can install just the set of things they need to run the specific commands they need to without worrying that there is a runtime dependency conflict.

To make things clearer, here are some commands defined that a user would run to freeze dependencies:

pip-compile -r --no-emit-index-url --output-file requirements.txt requirements/requirements.in
pip-compile -r --no-emit-index-url --output-file requirements-dev-unit.txt requirements/requirements.in requirements/requirements-dev.in requirements/requirements-dev-unit.in
pip-compile -r --no-emit-index-url --output-file requirements-dev.txt requirements/requirements.in requirements/requirements-dev.in
pip-compile -r --no-emit-index-url --output-file requirements-dev-dbconnect.txt requirements/requirements.in requirements/requirements-dev.in requirements/requirements-dev-dbconnect.in

[jurre]

I would be interested in knowing how the python bumping code is supposed to deal with situations where there are multiple requirements.in and requirements.txt files. Is this just not supported?

Yeah I'm pretty sure that it's currently not supported. Do you think there's a good deterministic way that Dependabot should handle it? I suppose we could try to find matching names between .txt and .in, but I don't think we know how to handle multiple files. Going to think about this for a bit, thanks for raising it 👍

[JimNero009]

Yeah, I can see two 'obvious' ways forward I suppose -- the first would be to somehow provide that info to dependabot at runtime (this might be something that is an addition to, rather than a core feature of, dependabot, but the core would still have to be able to expose a way to accept that configuration) and the second would be to assume a convention where the .in and .txt files directly match. Both have their downsides...

[jeffwidman]

I doubt we'll tackle this anytime soon. We'll probably tackle #3290 instead, as that's what the Python community seems to be moving toward (at least that's my understanding).