Tame the PyPI / Conda mapping chaos

[maresb]

I'm aware of three sources of truth for the correspondence between PyPI and Conda names.

1. Grayskull's config.yaml. Compact but maintained by hand and incomplete.
2. The so-called Grayskull PyPI mapping that's not actually used by Grayskull, but is used by conda-lock. This is complete, and uses some cool graph-theoretic heuristics to guess the best mapping. It's also possible to add an override in cases where the result is not as desired. (This might also be used by some parts of the conda-forge infrastructure.)
3. Prefix-dev's parselmouth. I am aware of its existence but I haven't yet had the chance to look at it in detail.

In order to reduce duplication, it would be nice to consolidate some of these, and especially to uniformize the edge cases. I'm very curious what @nichmor and @mariusvniekerk think.

[marcelotrevisani]

I am happy to unify it and have a final decision over it, that mapping for grayskull was created before all of those tools had that (I think) and it is being populated when necessary. But if there is something else better I am happy with that, I just don't want to add conda-lock as a dependency to grayskull. Maybe we can put that mapping somewhere else and create a virtual package with all the mapping? that should be interesting and conda-lock and grayskull can have a dependency in that package and we all going to have the same data.

[maresb]

I just don't want to add conda-lock as a dependency to grayskull.

Definitely! 😂

For example, one sane option would be to simply vendor the particular lookup module from conda-lock. Indeed, I noticed that conda-lock's code to parse Poetry-style caret and tilde dependencies was broken, so I grabbed Grayskull's solution in conda/conda-lock#679.

But maybe parselmouth is better, in which case we should figure out how to use that.

[BastianZim]

Just my 2 cents. I would recommend having a GitHub repo under an organisation that makes sense to host it, where a file is stored for everyone to download. That would have three advantages:

1. Can be downloaded quickly
2. Easy to add info, even if someone is not familiar with a particular tool as it would be tool agnostic (No need to learn a codebase of an app)
3. GitHub actions would allow us to create automated mappings without relying on a particular tool. If conda-lock is the best for now, use that. If parselmouth is better, switch over to that.

I'm just a big fan of separation because it has a smaller hurdle for onboarding and everyone can consume it.

[marcelotrevisani]

yeah, that is a good idea

[jakirkham]

cc @beckermr (who would know where the bot keeps this info as well)

[beckermr]

We supply the bot's mapping behind apis here in the conda-forge-metadata package: https://github.com/conda-forge/conda-forge-metadata/blob/main/conda_forge_metadata/autotick_bot/pypi_to_conda.py

The conda-forge-metadata package is distributed on conda-forge and pypi.

Use it if you like, but no pressure from me.

Even if you decide on something official, the bot's mapping will still be kept here as an api. It lives under the autotick_bot submodule and so should be able to peacefully coexist without confusion.