New local-recipes-index feature

[memsharded]

Changelog: Omit
Docs: Omit

Not documenting the feature until the blog post that announces it

• Feature: Allow adding a folder with a clone of conan-center-index or same folder structure as a remote.

How to use it:

$ git clone git@github.com:conan-io/conan-center-index
$ conan remote add c3ifork file:///path/to/cloned/conan-center-index
// use it as normal
$ conan search opencv*
$ conan graph info --requires=opencv/4.5.5 -r=c3ifork --format=html > file.html

[AbrilRBS]

Right now running conan list "*:*" -r=local for the current head of cci fails because it finds Conan 1-only recipes and does not get past that. A try-catch in the select() method inner loop of subapi/list.py might be enough to skip those cases?

[memsharded]

Right now running conan list ":" -r=local for the current head of cci fails because it finds Conan 1-only recipes and does not get past that. A try-catch in the select() method inner loop of subapi/list.py might be enough to skip those cases?

I have improved that by running a quick check of from conans in recipes, seems to be enough atm. I am also caching the result of the export to avoid re-exporting all the time.

[AbrilRBS]

To recap, some alternatives that might be nice to look into:

• conan remote add-local /home/barbarian/conan-center-index/
• conan remote add /home/barbarian/conan-center-index/ --local
• conan remote add /home/barbarian/conan-center-index/, @czoido suggests that if no "real" remote can have the protocol missing, there's no overlap between /absolute/paths and http(s)

I think the UX feels better this way, and less confusion about protocols
(@czoido justo had the same issue about the number of / after the file: I experienced last week)

[czoido]

This is supercool, my only comments are about changing the UX a bit, I would vote for adding an argument or not adding anything and distinguish between them with the http:

[memsharded]

Lets add a explicit argument, just in case in the future we want to support the flow clone + use, and then we need a URL

[memsharded]

Added conan remote add /path/to/repo --type=local to allow future flexibility and have full command irrespective of url input arg.

Name --type and "local" can be discussed, of course.

[tsondergaard]

Instead of requiring a local clone in the filesystem, could this allow me to specify just a git URL? For example:

conan remote add tsondergaard-cci https://github.com/tsondergaard/conan-center-index.git

With this setup can I have recipes in git and binaries in Artifactory?

[memsharded]

Instead of requiring a local clone in the filesystem, could this allow me to specify just a git URL? For example:

We have considered, even implemented it, and reverted. A couple of reasons:

• Fetching always the "head" and using it is not that great. Users want stability, being able to use a specific tag, branch or commit in many many cases. And that required to invent a whole DSL to define it: conan remote add repo git-url --tag=mytag and the like, really dirty thing
• Another very important use case, probably the most important is also being able to locally edit the recipes, and have the changes immediately available. This require a "user space" clone that users have simple access. If conan remote adds git urls, then those clones need to be in a private Conan folder, Conan can't be doing clones in user space.
• The control and semantics of updating/pulling/merging is also lost, or Conan would also need to invent yet another DSL around remotes like conan remote myremote --pull

So it was just like creating a more limited, poorer layer over git repos and commands, while also not allowing users to modify sources, and it was dropped, not really worth it. Looks like convenient at first sight, but in practice it doesn't work great for majority of cases.

With this setup can I have recipes in git and binaries in Artifactory?

Yes, Conan 2 binary management allows to use recipes from one repo (including this new local-recipes-index one) and binaries from other repos, because using revisions everywhere to guarantee consistency it allows to do so and guarantee that the binaries from one repo belongs to the same source recipe (recipe revision). Note that when binaries are uploaded to Artifactory, the recipe is also uploaded there together with the binary, so once this happens, it is no longer needed to use both and users can also get recipes and binaries from server repo too. This would be a very typical setup, and we already have many users that are doing this approach, even without the local-recipes-index: they work with a fork of conan-center-index, build their binaries from there and upload to their servers, and their developers and CI are just "consumers" of their own recipes+binaries. This is the approach documented in: https://docs.conan.io/2/devops/using_conancenter.html

[tsondergaard]

Vcpkg suggests consuming projects add vcpkg as a git submodule. Would that pattern work with the mechanism you are building here? Let's say my company maintains a fork of conan-center-index at https://github.com/acme/conan-index.git that is extended also to contain our own libraries. Could consuming applications then add that as a git submodule and use that as the repository as the source registry and then artifactory would only be used for binary packages?

[memsharded]

Would that pattern work with the mechanism you are building here?

Yes, this would work with this new local-recipes-index repo too, it uses a very similar approach. Nevertheless, it is possible that this approach is not the best in all scenarios, depending on the needs.

that is extended also to contain our own libraries

The local-recipes-index is intended for third party libraries. For packaging your own libraries, the approach of putting the conanfile.py directly in the repo might work much better, specially if you need to maintain different versions of your packages, because in that way the recipe evolves in parallel to the code, and it enables simpler flows like git clone + conan install + cmake, with just one single repo, no submodules. Git submodules management is not that straightforward and has issues too.