Support requirements.txt and setup.py as input

[sschuberth]

Is your feature request related to a problem? Please describe.

I'm looking at using thoth-solver in the ORT Analyzer as a replacement for the the custom scripts that we currently use for Python project dependency analysis. As the ORT Analyzer identifies the used package managers by the presence of what we call "definition files", we need to directly operate on requirements.txt or setup.py files in a code base.

Describe the solution you'd like

It would be nice if the thoth-solver CLI had new options to point at a requirements.txt or setup.py files to take the (direct) dependencies from.

Describe alternatives you've considered

I've considered parsing requirements.txt or setup.py ourselves in ORT, and then passing the direct dependencies to thoth-solver, but that seems error prone, and parsing of these files is better done in a project that's itself written in Python.

Additional context

N/A.

[fridex]

Thanks for the RFE.

Thinking of setup.py, do you publish the package with setup.py on an index? In that case, the solver can be pointed to your index and analyze it. If you want to extract dependencies directly out of a setup.py, that might be more tricky.

I can imagine a script that parses requirements.txt and passes information to the solver. It could use pip's internal api (not very nice, but can work):

>>> from pip._internal.network.session import PipSession
>>> from pip._internal.req.req_file import parse_requirements
>>> [i.requirement for i in parse_requirements("requirements.txt", PipSession())]
['requests', 'click']

[sschuberth]

do you publish the package with setup.py on an index?

Mostly not. ORT is regularly used to perform Open Source compliance checks on proprietary source code that is not published anywhere.

[sschuberth]

If you want to extract dependencies directly out of a setup.py, that might be more tricky.

I'm not a Python guy, but these guys seem to be discussing a new API for that.

[fridex]

If you want to extract dependencies directly out of a setup.py, that might be more tricky.

I'm not a Python guy, but these guys seem to be discussing a new API for that.

A solution would require executing setup.py in a target environment to get the dependency listing as there can be basically coded any requirements.

You can also create a virtual environment and install the given package into the virtual environment (temporary), ex:

$ python3 -m venv venv
$ sh venv/bin/activate
$ pip install .  # --no-deps might be a good option here

After that, you can ask what are the dependencies of the given package - using importlib metadata could be appropriate (as discussed in the linked issue).

This is out of our scope so I'm not in favor of making changes in thoth-solver directly (and maintaining the solution on our end). It is specific to your use case. Nevertheless, if there would be identified overlap with efforts, we are happy to support you even in our codebase.

[sschuberth]

You can also create a virtual environment and install the given package [...] After that, you can ask what are the dependencies of the given package

My main use-case is that I want to get the whole dependency tree without actually installing any of it.

This is out of our scope so I'm not in favor of making changes in thoth-solver directly

That's perfectly fine, I just wanted to check 😀 I guess it's better then for me to wait for the merge of pypa/pip#10748, that I discovered meanwhile. Looks like that'll do exactly what I'm looking for.

[goern]

/sig user-experience

[sesheta]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[goern]

/remove-lifecycle stale
/priority backlog

[Gkrumbach07]

Check if this can be covered by pip install --dry-run --report, if not we can look into this feature request more. An already available feature is to use thamos to run an advise on a pipfile/requirements.txt which will return a json document including a list of nodes and edges of the dependency tree solved.

[Gkrumbach07]

Can run pip install -r requirements.txt --dry-run --report report.txt and get the following output. Specifically requires_dist param can be used to create a graph from the list of installs.

/close

{
  "version": "1",
  "pip_version": "23.0",
  "install": [
    {
      "download_info": {
        "url": "https://files.pythonhosted.org/packages/43/b0/b916172eee4e946dea7155ed969865c1b2c01c883101e33d1eb0c224a6a0/daiquiri-3.2.1-py3-none-any.whl",
        "archive_info": {
          "hash": "sha256=b797a7ac94219dc26ef8ebf04f1f507eefa83a7d174e9eb41acc33e3ebf16f38",
          "hashes": {
            "sha256": "b797a7ac94219dc26ef8ebf04f1f507eefa83a7d174e9eb41acc33e3ebf16f38"
          }
        }
      },
      "is_direct": false,
      "requested": true,
      "metadata": {
        "metadata_version": "2.1",
        "name": "daiquiri",
        "version": "3.2.1",
        "summary": "Library to configure Python logging easily",
        "description_content_type": "text/x-rst",
        "home_page": "https://github.com/Mergifyio/daiquiri",
        "author": "Julien Danjou",
        "author_email": "julien@danjou.info",
        "license": "Apache 2.0",
        "classifier": [
          "Intended Audience :: Information Technology",
          "Intended Audience :: System Administrators",
          "License :: OSI Approved :: Apache Software License",
          "Operating System :: POSIX :: Linux",
          "Programming Language :: Python",
          "Programming Language :: Python :: 3",
          "Programming Language :: Python :: 3.7",
          "Programming Language :: Python :: 3.8",
          "Programming Language :: Python :: 3.9",
          "Programming Language :: Python :: 3.10"
        ],
        "requires_dist": [
          "python-json-logger",
          "systemd-python (>=234) ; extra == 'systemd'",
          "pytest ; extra == 'test'"
        ],
        "provides_extra": [
          "systemd",
          "test"
        ],
        "description": "..."
      }
    },
....
],
"environment": {
    "implementation_name": "cpython",
    "implementation_version": "3.8.10",
    "os_name": "posix",
    "platform_machine": "x86_64",
    "platform_release": "22.3.0",
    "platform_system": "Darwin",
    "platform_version": "Darwin Kernel Version 22.3.0: Thu Jan  5 20:53:49 PST 2023; root:xnu-8792.81.2~2/RELEASE_X86_64",
    "python_full_version": "3.8.10",
    "platform_python_implementation": "CPython",
    "python_version": "3.8",
    "sys_platform": "darwin"
  }

[sesheta]

@Gkrumbach07: Closing this issue.

In response to this:

Can run pip install -r requirements.txt --dry-run --report report.txt and get the following output. Specifically requires_dist param can be used to create a graph from the list of installs.

/close

{
 "version": "1",
 "pip_version": "23.0",
 "install": [
   {
     "download_info": {
       "url": "https://files.pythonhosted.org/packages/43/b0/b916172eee4e946dea7155ed969865c1b2c01c883101e33d1eb0c224a6a0/daiquiri-3.2.1-py3-none-any.whl",
       "archive_info": {
         "hash": "sha256=b797a7ac94219dc26ef8ebf04f1f507eefa83a7d174e9eb41acc33e3ebf16f38",
         "hashes": {
           "sha256": "b797a7ac94219dc26ef8ebf04f1f507eefa83a7d174e9eb41acc33e3ebf16f38"
         }
       }
     },
     "is_direct": false,
     "requested": true,
     "metadata": {
       "metadata_version": "2.1",
       "name": "daiquiri",
       "version": "3.2.1",
       "summary": "Library to configure Python logging easily",
       "description_content_type": "text/x-rst",
       "home_page": "https://github.com/Mergifyio/daiquiri",
       "author": "Julien Danjou",
       "author_email": "julien@danjou.info",
       "license": "Apache 2.0",
       "classifier": [
         "Intended Audience :: Information Technology",
         "Intended Audience :: System Administrators",
         "License :: OSI Approved :: Apache Software License",
         "Operating System :: POSIX :: Linux",
         "Programming Language :: Python",
         "Programming Language :: Python :: 3",
         "Programming Language :: Python :: 3.7",
         "Programming Language :: Python :: 3.8",
         "Programming Language :: Python :: 3.9",
         "Programming Language :: Python :: 3.10"
       ],
       "requires_dist": [
         "python-json-logger",
         "systemd-python (>=234) ; extra == 'systemd'",
         "pytest ; extra == 'test'"
       ],
       "provides_extra": [
         "systemd",
         "test"
       ],
       "description": "..."
     }
   },
....
],
"environment": {
   "implementation_name": "cpython",
   "implementation_version": "3.8.10",
   "os_name": "posix",
   "platform_machine": "x86_64",
   "platform_release": "22.3.0",
   "platform_system": "Darwin",
   "platform_version": "Darwin Kernel Version 22.3.0: Thu Jan  5 20:53:49 PST 2023; root:xnu-8792.81.2~2/RELEASE_X86_64",
   "python_full_version": "3.8.10",
   "platform_python_implementation": "CPython",
   "python_version": "3.8",
   "sys_platform": "darwin"
 }

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.