Deploy thirdparties in conan 2.X

[tatraian]

Hello,

In our project I have to ship our thirdparties together with our libraries (we develop an SDK like product, and we have to avoid thirdparty library version missmatches).

Our conan thirdparties comes from our conanfile.py with the running of conan install. In the conan file we use CMakeDeps generator. We use cmake to build our project after conan install ran. So I plan to ship the whole "conan install directory" to customers.

Since conan install directory contains only links to the local cache I used the two built in deployer of conan (direct and full deploys). For me (with 2.0 and 2.0.1) any of them worked perfect. In both cases the xxx_PACKAGE_FOLDER_RELEASE directory points to the deployed folder, but all paths in the xxx_BUILD_MODULES_PATHS_RELEASE variable still points to the local cache.

But independently the wrong BUILD_MODULES_PATHS variable the generated cmake variables are not relocatable, since absolute paths are used.

I tried to write a custom deployer and tried to change the deploy folder (dep.set_deploy_folder(new_folder)) to ${CMAKE_CURRENT_LIST_DIR} but that caused error in install since (this is logical) that is not a real directory.

Is there any way to make the cmake config files relocatable in the deploy folder? I think if someone call deploy on its dependencies than probably want a hierarchy without absolute paths.

I also attach a sample project too. I called it with command:
conan install . --output-folder install --profile:build ./gcc2004 --profile:host ./gcc2004 --build=missing --settings=build_type=Release -r conancenter --deploy=direct_deploy
conan_test.zip

thanks

[memsharded]

Hi @tatraian

Thanks for your question.

Indeed the cmake_build_modules property is not being relocated, I am already working on a fix for it, for next 2.0.3
I think I will also manage to make the root xxx_PACKAGE_FOLDER a relative path to the current .cmake file, making the final deployed path-independent.

However I have a couple of questions:

• This approach only works for 1 platform, because generated CMake files only have variability over Debug/Release/etc configs, but not over different OSs. How do you plan to deal with this? A different output-folder?
• You also need to be sure that executables are relocatable too, like no absolute RPATHs, or anything like that. Are you taking this into account too?

[memsharded]

I am submitting #13526 that would solve these issues, for Conan 2.0.3 release.

[tatraian]

Hello @memsharded

Thanks your very fast reaction, I will check it in 2.0.3.

We ship different binary packages on different platforms. So we have a binary package for windows and for linux. We don't plan to ship our thirdparties as an extra package, but together with our binaries. Conan/CMake are able to handle different build types on the same platform, but we use only the release version of the thirdparties (in both cases, during development and on the shipped system too). Since we write application that becomes very slow in "real" debug builds, we use "debug" build that is similar to RelWithDebInfo, just with O0 (on Windows we also link to Multithreaded Dynamic runtime as in case of release builds).

We basically ship a server with plugin support. Our server depends our libraries and the thirdparties too. The shared libraries are copied beside the exe on windows with cmake's install command (install(TARGETS ... RUNTIME_DEPENDENCIES)), on linux the same happens just into the lib directory. On windows this is enough to run the exe, but on linux we have play with RPATHs or LD_LIBRARY_PATH environment varible here. But thirdparties (as a directory) is copied into the install directory too, so the shared libararies will be duplicated on the disk (yes, this is ugly). When someone writes a plugin it will do it with a find_package() to our project. Our custom config.cmake file calls find_package to all of our thirdparties (the same cmake file is copied into the install that is used during development). In this case the plugin will find the shared libraries in the install/thirdparties directory, but that are exactly the same that are around the exe on windows or in the lib dir in linux.

Anyway an alternative solution could be to use conan instead of "installed" binaries and headers, since the solution above basically makes impossible the using conan in plugin developments, since (as far as I know) there is no way to add prebuild libraries to conan project as dependency (so for example conanfile dependency description about asio library use our shared version instead of its own dependency resolution).
But this solution can work just only with shipping our conan packages with conan server to customers, since we use custom recipes.

But you see well, I really don't think this solution is good, or easy. Unfortunately our sdk word is the most complicated that I had to work with, and I didn't really had better idea(s) as above...

[memsharded]

Anyway an alternative solution could be to use conan instead of "installed" binaries and headers, since the solution above basically makes impossible the using conan in plugin developments, since (as far as I know) there is no way to add prebuild libraries to conan project as dependency (so for example conanfile dependency description about asio library use our shared version instead of its own dependency resolution).

Conan 2.0 made this possible, but only for tool_requires, you can check https://docs.conan.io/2/reference/config_files/profiles.html#system-tools
But a general replacement of libraries it is practically impossible to make it work smoothly in practice, the way information is propagated in the graph really needs meta-information contained in the recipe, and just some binaries in disk will not make it.

Thanks your very fast reaction, I will check it in 2.0.3.

yes, in any case, I hope the features above in the PR could help you to create the deployed binaries that could be convenient for your distribution. Just a couple of warnings when you test it:

• Recall that generated .cmake files are not multi-OS, you need to use different output-folders
• The VirtualBuildEnv and VirtualRunEnv seems way more complicated to make the paths relative in Linux (managed to do it in Windows). It means that the environment files will not be relocatable in Linux/Mac. In practice this can be solved with a search&replace of a string in the generated .sh files. Not sure if you need the environment too, or with the cmake files you are good.

[memsharded]

#13526 merged, it will be in next 2.0.3, looking forward your feedback, thanks!

[memsharded]

There is still a gap in this, because for cmake_layout the generators folder will not be a children folder, and then paths will not be made relative.
I will try to investigate this for 2.0.4

[memsharded]

This was finally fixed in 2.0.4, at least CMakeDeps and CMakeToolchain integration. And environment scripts for Windows .bat files (the other OSs are a bit more challenging, will wait for user demand).

Preparing some example and documentation in conan-io/examples2#94, I think this can be closed in 2.0.4 milestone, feedback welcome!

[tatraian]

Hello @memsharded,

Sorry for the late reaction, bu I was lost my work (this issue came during moving our project a totally new build infrastructure with deadline). I checked the fix with direct_deploy on 2.0.4 and it works smoothly!

Thanks your very fast fix and sorry again may reaction delay...

[memsharded]

No prob @tatraian !!

We know that these things happen (also happen to us, don't hesitate to comment or ping on an issue that didn't get a response, to re-surface it), no need to apologize for it, thanks very much for following up and reporting :)