[question] Projects with an in-source recipe

[Minimonium]

What's the strategy for the inclusion of such projects? For example - https://github.com/catchorg/Catch2

[uilianries]

I would say keep all them as third-party and deprecate in-source recipes.

pros:

• author don't need to maintain the recipe/ci/upload
• the project don't need to receive an extra step on their ci script (travis, ...)
• any issue related to the recipe/conan the author don't need to care about or look for help

cons:

• the author can't manage the recipe as owner (less power)
• the license won't follow the same from the project
• the author will need to observe an external project to know about the recipe

[Minimonium]

There is one moment of transitive Conan dependencies. For projects that don't have dependencies having a fully separate recipe is okay. But what about projects that use other Conan projects? They'd like to have the recipe in-source.
Would the advice be to submodule the recipe, for example?

[uilianries]

As these new packages are uploaded to Conan Center, the author will need to update the recipe only, with new references.

If some author want to keep in-source recipe, I see no problem. However, someone will need to maintain the recipe.

[Minimonium]

The problem here is the duplication of the recipe, which is unfortunate. I think it's worth it to think about it. 🤔

[ghost]

I think having 2nd in-source recipe is useful for "git" versions and maybe other repositories.

[Minimonium]

Related conan-io/conan#4734

[KerstinKeller]

I feel a bit ambiguous about the in source vs out of source recipe topic.
But I looked at the structure of the repo (because internally we keep almost the same structure), and the build logic for Jenkins. Unfortunately that one is hidden in a library 😞

Anyways, depending on how the build logic works, it could certainly be suitable to integrate libraries with an insource recipe by specifying urls instead of folders in the config.yml file:

Take the boost example:

versions:
    1.70.0:
      folder: all
    1.71.0:
      folder: all

If boost had an insource recipe you could do:

versions:
    1.70.0:
      url: https://github.com/boostorg/boost/commit/8f9a1cf1d15d262e09c16a305034d8bc1e39aca2
    1.71.0:
      url:: https://github.com/boostorg/boost/commit/afb333b7c5101041f0280b2edf155c55114c9c95

And the job builder would checkout that commit and continue from there.

[SSE4]

the author can't manage the recipe as owner (less power)

this is questionable argument, as author may decide to force push to his repository, or even remove it completely, but to keep packages stable and always available it might not be desired.

[uilianries]

the author can't manage the recipe as owner (less power)

this is questionable argument, as author may decide to force push to his repository, or even remove it completely, but to keep packages stable and always available it might not be desired.

I mean, the author is not able to force a push here in conan-center-index, or use any admin power.

[SergiusTheBest]

Can we make an out-of-source recipe that references in-source recipe?

[uilianries]

Do you mean alias ?

[SergiusTheBest]

@uilianries It looks very close if it could reference a github repo instead of another package.

[Minimonium]

So, friends, to sum up.

Why the out-of-source strategy for the Index important:

• Any patches could be applied to all recipes at once for best practices in a unified manner. Repo maintainers are responsible for the quality of recipes.
• Any security updates could be applied to everything at once too.
• Any contribution process is visible to all and can get help from the community in a simple manner.
• Importantly, Conan plans breakages (Conan 2.0). The Index allows migrating the whole repo at once.
• Library authors don't need to know about the recipe or the Index.

Moments to think about:

• Release is bottlenecked. The process of publishing the new version after the upstream requires a manual effort and audit. I'm sure it's not an unsolvable problem, there are ways to work around it. For example, granting the library owners the fast-track rights or maybe making something with the Github API to allows authors to add a command into their CI to automatically make a PR with a new version. Would be pretty cool.
• Duplication. I think that's a problem only for projects that use Conan as a vehicle to get dependencies. So the conanfile.txt and the same functions of the conanfile.py. Right now, the whole machinery for calling the source(), build(), package() is used to make a package. My opinion on it is that the upstream package should back-port all best practices from the Index, which means that leaving the window for referencing the out-of-Index recipe compromises the quality of the recipe. More at In-source vs. out-of-source recipes conan#4734 (comment)

The out-of-source way is clearly the way to go for the long-term maintainability of the repository. The goal is to figure out how to find a way to solve some shortcomings with it.
It would be good to have some understanding of what's Conan's plan in the near future to address moments with it.

[ghost]

* Library **authors don't need to know** about the recipe or the Index.

Doesn't that kinda defeats the point? For my libraries Conan is the only supported way to consume them because Conan is the only sane package manager. Using anything else is disaster.

[danimtb]

Generally, I agree with the summary of @Minimonium and believe this repository and build service is the best choice to have a strong conan-center and better community. Not to say that the contribution process is transparent and the curation is done community wise.

On the other hand, we knew the in-source approach was going to be compromised in the early stages and authors having the recipes in their projects (like @Lyberta) could end up with a duplicated recipe.

However, this is something we will solve in the near future and approaches like the one proposed by @KerstinKeller could fit nicely. Will need to look into this issue and discuss different approaches further so we come up with the right solution for the different scenarios.

[Minimonium]

@Lyberta The point is to have a large number of libraries to consumers. If authors want to engage with the Conan community - great. If they don't - doesn't matter. 🙂

[ghost]

@Minimonium And what if they use other horrible package manager such as APT, DNF, Pacman, vcpkg?

[Minimonium]

@Lyberta We migrate their dependencies and patch them out of their hard bindings to specific package managers if needed. But it's not useful to fight windmills. 🙂

[Minimonium]

Users brought up an additional use case of when in-source recipes are important: workspaces
skypjack/entt#380 (comment)

Removing the recipe from EnTT repo will make it a bit cumbersome for me to test things eg. the experimental or other branches.

Having the recipe there allows me to add the repo as a part of a Conan workspace, without it I can't do that.

What can we do about it?

[memsharded]

Thanks for this new bit of information.

I see the issue, but I think that at the moment, I would still recommend keeping an in-source recipe for development, while submitting a recipe for conan-center-index. It might be a bit of duplicate work sometimes, but to summarize some of the above, it also helps in other ways:

• it is possible to have a very simple recipe, e.g. for header-only packages for conan-center-index, while keeping a more complicated one that can run tests and other stuff (like workspaces) in-source.
• can streamline the contribution of many versions at the same time, and specially to backport package fixes to all versions very easily, highly simplifying the work of contributors and collaborators.

Thanks for the feedback, even if it seems difficult to solve the issues for both worlds at the same time, we keep these things in mind.

[Croydon]

While I would also like the idea that projects keep a conanfile in-source for dev versions, I think that reality is slowly catching up.

For people better familiar with Conan it might be clear which Conan recipe you want to use, when both CCI and in-source recipes exist, but for the wast majority of people this seems to be confusing. People start reporting bugs in which it isn't clear what they actually use. Even worse when the recipes diverge from each other. Maintenance and support is hell. (I have realized this e.g. with Google Benchmark)

Some upstream projects also removed their own conanfile and now refer to CCI. Latest example is probably Paho, see #1395

I also can't think of any solution to refer/link from CCI to an in-source recipe that would really work. Working across so many repositories, platforms, systems, workflows to contribute would be living hell too. And "quick fixes" would probably take pretty long.

Personally, my current point of view is, that in-source recipes only pay off if the project itself is so specific that there is no general interest in packaging it, or if legal reason don't allow to add it CCI or something exceptional along those lines. In any other case, adding a recipe only to CCI is probably much easier to discover, maintain and support.

[Nipheris]

Hi ConanCenter crew,

I have thought a lot about this since this issue was created, and I'm bound to acknowledge I see no promising future with the current approach. And here is why.

Both Conan team and community understands, that Conan recipe is something like a lifecycle wrapper for the whole C++ library/tool. With Conan we finally have some standardized interface to build, package and consume any C++ library. No more reading READMEs to build the library. No more tricky environment configuration. No more 8 hours spent to prepare the build system you never heard before, and all the dependencies. For this reason, I find silly and unacceptable to split Conan recipe and all the rest source code and build scripts (in terms of VCS repos, versioning and release/publish cycle).

Of course, out-of-source recipe is a necessary evil to bring existing open-source library into Conan community: obviously because majors like ICU or Boost or Qt are not interested in Conan and its community before it becomes mature, and the community will not become mature before we make major libraries available through Conan. But I think it is like "find modules" in CMake: they were the necessary thing to be implemented, but today providing CMake configs by the library authors is a de-facto standard, and a lot of libraries follow it. The same for Conan: the more C++ projects with first-class Conan support we have (with in-source recipes), the more stable and fast-evolving packages we have.

But the current out-of-source approach of ConanCenter runs against this. For some technical reasons I should copy my in-source recipe in CCI, or make an alternative one and open a PR. For every version of my library. For some reasons we assume it is enough to review the recipe itself, without reviewing the rest of source code or, at least, build scripts or build definitions. Do we really know the things better than library authors?

It is ironic, but is seems the projects NOT interested in Conan ecosystem have even BETTER experience: it is only one recipe in CCI, supported by community. The project maintainers should do nothing to support it, because out-of-source workflow of ConanCenter works perfect in this case. But if I want to introduce Conan in MY project and use it to manage lifecycle and dependencies, I have hard time to decide what to do with my in-source recipe to make it available in CC. No npm run build + npm publish like with NPM. No cargo publish one-liner. In fact, I'm punished for usage of Conan + ConanCenter 😃

Why can't we do something like Cargo does? To allow project maintainers to publish the source code package to some registry (WITHOUT BINARIES at all), and prebuild all the binary packages for supported configurations like now?

We can even try to keep the current workflow with conan-center-index repo and PRs triggering the builds by using some lightweight "pointers" in conandata.yml instead of real recipe files (like Conan already does with scm attribute when scm_to_conandata is turned on, I think it is nearly what @KerstinKeller is talking about).

And it is also obvious that current approach does not scale. Just imagine if all the NPM packages published to npmjs.org are passed throught review-accept process like CCI have. It is hilarious.

[Minimonium]

@Nipheris Hi, Stanislav!

It's been almost two years since the start of the CCI project so at this point I dare to say we have a vague idea of the benefits of such a solution.

I agree with you, that the ultimate goal could be to allow users to instantly publish and consume their solutions, achieve an orchestration level to the similar level as NPM or Cargo. But, to answer your question:

Why can't we do something like Cargo does?

The issue is that C++ is 30 years old and it never had (and still doesn't have) a unified build structure. At this point, it's already too late to change it. The biggest challenge of modern C++ package managers isn't to design a fancy scheme, it's not about the command-line interface and it's not about a file layout. It's about thousands of edge cases of how software has been built and integrated through decades.

C++ is not a green field language, its importance is only in the ability to use the accumulated knowledge of existing libraries. If a tool can't pull in an existing library - it's a useless tool for most developers.

We must understand that Conan is not truly stable yet, it's under extremely active development and not because developers are sloppy, but simply because C++ is rich with use cases and CCI allows to dig up them one by one in the tremendous effort to apply Conan scheme to existing libraries.

It means that Conan changes and it'd be required for library authors to timely update all of their recipes for all of their project versions which is logistically impossible. But due to the monumental efforts of maintainers, CCI can push all of the contained recipes, for all versions of all dependencies, to new features introduced to solve integration problems.

It also explains why CCI is mostly interested in reviews of the recipes, to enforce the most recent best practices and recommendations. Up until the point where maintainers can say that there isn't much to review in most PRs - I don't see how it's possible to allow a more flexible approach to in-source recipes.