Add pyproject.toml option to prevent requests on pypi.org

[yjeand]

My corporate proxy does not allow access to pypi.org. We have a private mirror accessible, but unfortunately conda-lock attempts to make requests to pypi.org even when the private mirror is set up, resulting in a conda-lock exception.

With this CLI flag, we can ensure that no request goes to pypi.org.

[netlify]

✅ Deploy Preview for conda-lock ready!

Name	Link
🔨 Latest commit	5e9adfa
🔍 Latest deploy log	https://app.netlify.com/sites/conda-lock/deploys/63cf9af62bbffa0008320a05
😎 Deploy Preview	https://deploy-preview-304--conda-lock.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[maresb]

Thanks a lot for this!

Before we go ahead with this, I'm wondering if adding a new option is really necessary. (Such stuff adds complexity and long-term maintenance costs.) In particular, I am wondering why pypi.org is being queried in the first place if all the packages are discoverable through the private mirror. Perhaps it's possible to simply prevent this access in the first place.

Could you share the stack trace so that we can see the execution path leading up to the PyPI request?

[yjeand]

Thank you very much for your answer!

I agree that the ideal behavior would be to prevent unecessary requests to pypi.org without having to use an additional option. At first glance, the fact that pypi is the last item in the repositories list could be enough... but this is not what I see in my use case.

Here is an excerpt from my logs, including the commandline used and the stacktrace. Please note that I used conda-lock v1.2.1.
stacktrace.txt

Thank you for your time, please let me know if I can help in any way!

[maresb]

Thanks a lot @yjeand for the stack trace and your willingness to help!

We are currently using a vendored copy of Poetry which is v1.1.15. In your PR here you have identified the block where we initialize the Pool object with

pool = Pool(repositories=[*repos, pypi])

Based on your stack trace, it seems that the Pool object is searching all the provided repositories. On the other hand, it seems that there are ways to disable PyPI from Poetry. So I would ask: does your Poetry configuration allow you to prevent Poetry from accessing pypi.org? If so, is there a simple adjustment to the conda-lock code so that we respect this setting? For instance, I notice that poetry itself never specifies the repositories argument to the Pool constructor, so I suspect that manually adding pypi to repositories is something we shouldn't be doing by hand; instead we should use the correct Poetry mechanisms.

Would you be able to look into this?

[yjeand]

Thank you for this analysis. For my project we currently use Poetry for our linux builds (I'm working on conda-lock support for Windows, and maybe to replace Poetry for Linux down the road). I can confirm that on our Linux builds Poetry (used directly, outside conda-lock) does not attempt to access pypi since we have added a source with default = true in our pyproject.toml file. However, conda-lock does not seem to transmit this setting to the vendored version of Poetry.

I will take a look into it and report back here.

[yjeand]

Hello,

Here is a quick summary of my experimentations. No ideal solution has come up.

I think that the most natural behaviour for Poetry-based projects would be for conda-lock to respect the sources declared in pyproject.toml for Poetry. If a source has default = true then conda-lock could deactivate pypi.org (or let the Poetry solver do it). However at the moment conda-lock does not use those sources from pyproject.toml at all, and I do not feel capable of making such a big change. Thinking about it a bit more, this approach would be probably awkward for pyproject.toml-based projects which use a build system other than Poetry.

Conda-lock instead relies on globally defined sources in the user configuration file for Poetry, like explained here. And as far as I know, there is no way to flag a repo as default when using poetry config like this (I have tried manual modifications of the config file, it did not work). If we add a way to flag a repo as "default" in conda-lock, I feel like we are getting back to a solution roughly equivalent to the original --no-pypi flag of my pull request in terms of user impact.

Coming at it from another angle, I wonder if it would not be simpler to just test a pypi.org connection from conda-lock to decide if we add it to the repositiories list that is passed to the vendored Poetry solver. Something along the lines of:

    try:
        PyPiRepository(disable_cache=True).get_package_info("pip")
        repos.append(PyPiRepository())
    except RequestException as e:
        print(f"Could not access pypi.org, error was:\n{e}", file=sys.stderr)
    pool = Pool(repositories=[*repos])

in pypi_solver.solve_pypi()
(I'm not sure if the cache is reset each time conda-lock is run, so I deactivated it just in case)

What do you think about all this?

[maresb]

Thinking about it a bit more, this approach would be probably awkward for pyproject.toml-based projects which use a build system other than Poetry.

This is an excellent insight.

Perhaps it would make more sense to store conda-lock configuration related to project.toml in a [tool.conda-lock] section inside pyproject.toml itself. This would help keep the CLI clean. But I don't think we want to use it as any sort of primary configuration source, so this should somehow be self-evident. Maybe we could do

[tool.conda-lock.pyproject]
use-pypi = true

How do you feel about this idea?

[yjeand]

Good idea, it would certainly work for my project.
I will try to implement it, see how it goes.

[maresb]

For a subsequent PR, it might be interesting to deprecate --extras, --filter-extras, and --pypi_to_conda_lookup_file in favor of a similar pyproject.toml-based config.

[yjeand]

Here is my attempt at implementing the pyproject.toml-based config to disable pypi.org.
I have named the setting differently from your example because it seemed more natural to me, but feel free to change it.

[maresb]

This looks excellent @yjeand, thanks so much!!!

On second thought, I agree with you that [tool.conda-lock] is the correct level for this setting.

One final request: could you please add a short note to the README as documentation? After that, this looks ready from my perspective.

@mariusvniekerk, could you please have a look?

[yjeand]

I have added a couple notes in the README.