[bug] build_requirements causes entire recipe to be skipped

[Hash17]

Environment details

• Operating System+version: VMware workstation / Ubuntu 20,04
• Compiler+version:
• Conan version: 1.59
• Python version: 3.8.10

Steps to reproduce

1- in conan recipe, depend on a package using the following for a package called package_3 for example
def build_requirements(self):
self.tool_requires(f"package_1/0.6.0@user/{self._conan_channel}")
self.tool_requires("package_2/0.1.0@user/{self._conan_channel}")

this works completly fine and as expected. but if i change package_2 version to 0.2.0 , the entire recipe for package_3 is skipped and instead it is just downloaded from conan remote (from the logs in 1st case i can see it is showing a build requirements dependency on package_1 and package_2 then calls build and package, in 2nd case it doesnt show any of that and it directly logs that it is retrieving package_3 from conan artifactory)

Note_1: if i comment dependency on package_2, it considers the recipe and works as i expect it to do in case of package_2 being missing, i get another expected error due to missing package_2
Note_2: am building with -build=missing, if i remove it and add every error i get for example if i get error that missing prebuild for googletest , arm-clang and i add them --build=googletest --build=armclang , the build continues as i expect it

2 Questions here :
1- is such a behavior even possible that it skips the build_requirements and the entirerecipe and instead downloads from conan artifactory just because the package is there? i want to build it not install/download it.
2- for full disclosure, package_2 is build with a different profile, could that cause such an issue? i would expect an error maybe ?

Logs

No response

[memsharded]

Hi @Hash17

thanks for your question.

Some quick feedback:

• self._conan_channel is an internal, protected Conan variable. You shouldn't be using it in your recipes, it will eventually break (and it certainly breaks in Conan 2.0, that removed it)

this works completly fine and as expected. but if i change package_2 version to 0.2.0 , the entire recipe for package_3 is skipped and instead it is just downloaded from conan remote (from the logs in 1st case i can see it is showing a build requirements dependency on package_1 and package_2 then calls build and package, in 2nd case it doesnt show any of that and it directly logs that it is retrieving package_3 from conan artifactory)

1- is such a behavior even possible that it skips the build_requirements and the entirerecipe and instead downloads from conan artifactory just because the package is there? i want to build it not install/download it.

Yes, that is the behavior. If the package binary exists, it is downloaded, and the tool_requires are not necessary, because it doesn't need to build from source, and then it doesn't need the tool_requires

2- for full disclosure, package_2 is build with a different profile, could that cause such an issue? i would expect an error maybe ?

It doesn't matter. As long as the binary for package_3 exists, the tool_requires are not needed, it is irrelevant what profile do they define.

[Hash17]

Hi James,

You are doing an amazing job answering this fast and wirh such direct answer even though i know my description was a bit confusing (did my best). Thank you and i wish you a good evening/day

[memsharded]

You are welcome :)

Extra bonus:

• You need to make sure you are providing both build and host profiles in 1.X. Like -pr:b=default -pr:h=myprofile. This will be default in 2.0, but it is important to start using it in 1.X to be ready, and for things to be correct
• You might want to start having a look to the Conan 2.0 model, which has slightly changed wrt to tool_requires, they will be in the graph, but only the recipe, while the binary (the heavy thing) will be marked explicitly as Skip. It also contains some

[Hash17]

We are planning to migrate soon to 2.x hopefully, it's already in the backlog.

I read your answer again and i think i still have some confusion.

1- if what you say is correct (if the package exists, it's downloaded), then why if i remove --build=missing it throws an error instead of just downloading package_3?

2- is this because i only have a package method and no build? Because thats my case

What am trying to do is to package 2 conan packages, and i depend in them by tool_requires and in package stage i just copy both into final package

[memsharded]

The approach you are trying to do with tool_requires is most likely not good, because tool_requires() should be applications, that live in the "build" context. Please read the warnings in the docs, like https://docs.conan.io/1/devtools/build_requires.html

tool_requires are designed for packaging tools, utilities that only run at build-time, but are not part of the final binary code. Anything that is linked into consumer packages like all type of libraries (header only, static, shared) most likely are not tool_requires but regular requires. The only exception would be testing libraries and frameworks, as long as the tests are not included in the final package.

The thing is that in any scenario that the build profile is different from the host one, even if not cross-building, you will be packaging the wrong thing. The re-packaging use case will be considered in #13171, in the 2.X roadmap. At the moment you might be able to achieve such re-packaging via the client side, like doing a conan install of something, doing a collection of what needs to be be re-packaged in the generate() method with one recipe and then doing a conan export-pkg with another recipe.

Conan 2.0 will provide better ways to do this, with the require traits, like making requirements visible=False.

Otherwise why the use case for re-packaging? Why not depending directly on the dependencies in their original packages?

Regarding your first point, it is difficult to know, the whole flow is not very clear and it would be necessary more information, and more structured. The best is typically self-contained minimal reproducible examples, that is, some very simple recipes, mostly empty, or just packaging some dummy file, and a build.py script that runs the necessary steps to reproduce the case. Collaborating over real code makes things easier and faster.

[Hash17]

Okai i understand everything you said about build_requires, i can check the documentation for "generate" and "export-pkg" and see how they fit in. But my iae case case is different than the issue you referenced #13170 , let me explain:

Iam not building anything at all, package 1 produces a folder in its package folder, package 2 produces another folder. My final package simply wants both in a single package, it simply copies the folder from package 1 and package 2 into package 3, so i dont re-build package 1 or 2 at all, i only use build_requires (i understand thats not correct) so that they are present in my conan cache, and then in my package method i self.run to copy them with rsync (i do some manual work to get path to 2 packages).

I will try to prepare a dummy recipe to show my use case if this still doesnt help

[memsharded]

Yes, it clarifies a bit, and I think this is still the #13171 ticket (Note you linked a wrong ticket, off by one), which is totally the "re-package" concept. It doesn't matter if you build it or not, the re-package is basically doing a copy inside another package of the contents of some referenced dependencies.

Even if you don't build from source, and you are packaging pre-compiled binaries, they will have some "settings", some architecture, some binary compatibility that will match the "host" settings, and that can be different to the "build" settings of the current machine. It would be good to clarify what are you packaging.

This is doable both in 1.X and 2.0, but the problem is that fetching and downloading those dependencies will not be optimized in 1.X, and they will still be downloaded (you might try the private=True, but that has also its own issues). In Conan 2.0 is more doable with the traits.