Integrate with Conan package manager

[solvingj]

Per @waruqi request, opening this ticket to track an effort to add support for Conan packages.

Ideally, each build system would not need to support each package manager, but this is not a solved problem yet. Until then, custom support must be added to find Conan packages installed on the local machine.

[waruqi]

Ok, I will support it.

[yssource]

xmake is going to work as a generator in Conan, isn't it?

[waruqi]

@solvingj I have supported to find conan packages using lib.detect.find_package. You can update xmake from dev branch and try it.

$ xmake l lib.detect.find_package bzip2

{
    links = 
    {
        bz2
    }
,   linkdirs = 
    {
        /Users/ruki/.conan/data/bzip2/1.0.6/conan/stable/package/534dcc368c999e07e81f146b3466b8f656ef1f55/lib
    }
,   includedirs = 
    {
        /Users/ruki/.conan/data/bzip2/1.0.6/conan/stable/package/534dcc368c999e07e81f146b3466b8f656ef1f55/include
    }
}

Find and use conan packages in project with xmake.lua

target("test")
    set_kind("binary")
    add_files("src/*.c")
    on_load(function (target)
        import("lib.detect.find_package")
        target:add(find_package("bzip2"))
    end)

And add_requires also support to find conan packages:

add_requires("bzip2")
target("test")
    set_kind("binary")
    add_files("src/*.c") 
    add_packages("bzip2")

[waruqi]

@yssource Yes, we are planning to do it.

[solvingj]

Thanks! Can you tell me how it finds packages? In theory, the user should have to include the generated conanbuildinfo.lua file, and then cmake could look for the targets with the Conan- prefix. Like, Conan-zlib, although we can decide how the names will be formatted when we create the generator.

[waruqi]

@solvingj You can see 8062913

[solvingj]

Ok, I have looked at the code and unfortunately, the way you have tried to solve the problem will not work. I will try my best to explain the problem, and how we'll need to proceed.

Build System Settings
As you know, every time a build system such as xmake runs a build (compiles and links), it does so with a specific set of "settings". For example:

• MSVC 15 compiler, targeting x86 architecture, in Release mode, on Windows.
• GCC 8 compiler, targeting x86_64, in Debug mode, on Linux
• Etc...

Libraries generated with one set of these "settings" can generally only be linked into binaries being compiled with the same settings. Furthermore, compiler flags, preprocessor flags, and special library options might need to be different, depending on the current settings.

Also of note, developers often have to switch between different settings very frequently, going from Release to Debug, or from x86 to x64, etc. As a result, build systems generally create a unique folder structure for each unique combination of settings to hold the output files. Conan does something similar, but in a far more robust way. Also, it produces more than just folders with files, it produces critical variables for consuming build systems, such as defines, cflags, cppflags, libs, and more.

Conan Settings
Conan defines the following four default "settings", and each time a recipe is built and the package is stored, it's stored in a unique folder based on these settings:

• os
• arch
• compiler
• build_type

It does this each time Conan is run, by taking the values for all the settings (in the current execution) and hashing them to produce a unique_abi_hash.

Conan also lets packages define custom options for libraries, which will be factored into the unique_abi_hash, and thus be stored in separate folders. For example, many libraries add an option called shared, which lets users choose between the shared library binaries or the static library binaries.

Finally, as mentioned, each package might have advanced logic to choose variables that it will pass to consumers, including defines, cflags, cppflags, libs, and others. All of this is determined at execution time by a python function called package_info(). Here's the function defined by zlib package:

    def package_info(self):
        if self.options.minizip:
            self.cpp_info.libs.append('minizip')
            if self.options.shared:
                self.cpp_info.defines.append('MINIZIP_DLL')
        if self.settings.os == "Windows" and not tools.os_info.is_linux:
            if tools.os_info.is_windows:
                self.cpp_info.libs.append('zlib')
            else:
                self.cpp_info.libs.append('z')  # MSYS/Cygwin builds
        else:
            self.cpp_info.libs.append('z')

The Workflow of Integration
Due to these and other complexities, conan's generator system is the only way to consume conan packages from a build system like xmake. There can be some automation between xmake and Conan (link below), but in the end, xmake MUST work with the conan generator system by consuming it's files.

Manual Workflow
Here is an example of the process if conan is run manually, with no special "find_package" functionality added to xmake.
Note: this is intentionally verbose and explicit for demonstration:

1. User wants to build gbox with xmake using settings: Linux/gcc8/x64/Release.
2. User wants zlib static library from Conan.
3. User creates conanfile.txt in the root of the gbox project, and lists dependencies:

[generators]
xmake
[requires]
zlib/1.2.11@conan/stable

1. User creates a unique folder in some build directory for gbox, for the desired settings. Example:
   build/linux_gcc8_x64_release

2. User runs the following command:
   conan install . <username>/testing -s os=Linux -s compiler=gcc -s compiler.version=8 -s arch=x86_64 -s build_type=Release -o zlib:shared=False --install-folder="build/linux_gcc8_x64_release"

3. This produces the following file:
   build/linux_gcc8_x64_release/conanbuildinfo.xmake.lua
   Note: remember, the paths and variables are specific to the settings used (Linux/gcc8/x64/Release), and only containing the specific dependencies of gbox, as opposed to some global cache of all packages from the whole system.

4. The user adds a line to their main xmake build file, something like:
   if is_plat("linux") and is_arch("x86_64") and is_comp("gcc") and is_comp_ver("8") and is_mode("release") then
   add_include("build/$(platform)_$(compiler)$(compiler_version)_$(arch)_$(configuration)")

Now when the user runs the xmake build, it will have access to all the appropriate variables and values for zlib.

Improving the workflow

Obviously, the above example is extremely naive and unnecessarily verbose. I used it because it shows how all the variables are relevant in both xmake and conan, and it demonstrates the necessity of making sure both tools are using the same settings information all the time.

To improve the flow, there are several things that can be done.

Here's an integration for Cmake, implemented as a CMake module. With this approach, CMake did not have to be modified at all. Users simply have to include this module in every project which wants to use Conan. It has some disadvantages, but it's an option for xmake. At the very least, it will be helpful for you to look at to understand how to interact with Conan programatically.
https://github.com/conan-io/cmake-conan/blob/develop/conan.cmake#L29

However, since you're already writing custom code for conan directly into the xmake platform, here is another option, which is much more streamlined. You could add an environment variable or command-line-parameter to xmake. Something like:
XMAKE_CONAN_ENABLE/--conan-enabled

If the user sets or passes this option when running xmake then you can perform the following automatic steps:

• Search for conanfile.txt or conanfile.py in the project root (for starters)
• If found, execute conan install . , passing xmake settings to Conan (refer to conan.cmake)
• For --install-folder, pass xmakes current temporary build directory
• Automatically add_include() the generated conanbuildinfo.xmake.lua to the project
• For the find_library behavior, if the user wants zlib, search CONAN_ZLIB as an alias for zlib.

** Summary **
I think this last approach is the best. It uses Conan the way it's intended, and requires minimal code within xmake. It ensures xmake is always using the correct, and current Conan paths and variables for the current xmake execution, and takes advantage of xmake's unique directories to store generated files. This way, users don't have to manage unique directories just to hold generated files themselves (like in the manual example i gave).

I will start work on the xmake generator now. It should not take long.

[waruqi]

Ok, I will improve it.

[solvingj]

Great, thanks! See #331 for the working draft of the xmake_generator.

[waruqi]

I improved find_package and re-integrated conan, you can see (in dev branch) https://github.com/tboox/xmake/tree/dev/xmake/modules/package/manager/conan

add_requires("CONAN::zlib/1.2.11@conan/stable", {alias = "zlib", debug = true, 
    configs = {build_requires = "xmake_generator/0.1.0@bincrafters/testing", build = "all"}})

add_requires("CONAN::OpenSSL/1.0.2n@conan/stable", {alias = "openssl", 
    configs = {build_requires = "xmake_generator/0.1.0@bincrafters/testing",
    options = "OpenSSL:shared=True", build = "all"}})

target("test")
    set_kind("binary")
    add_files("src/*.c") 
    add_packages("openssl", "zlib")

Build project

ruki:test_package ruki$ xmake
checking for the architecture ... x86_64
checking for the Xcode directory ... /Applications/Xcode.app
checking for the SDK version of Xcode ... 10.14
note: try installing these packages (pass -y to skip confirm)?
  -> CONAN::zlib/1.2.11@conan/stable  (debug)
  -> CONAN::OpenSSL/1.0.2n@conan/stable  
please input: y (y/n)

  => installing CONAN::zlib/1.2.11@conan/stable .. ok
  => installing CONAN::OpenSSL/1.0.2n@conan/stable .. ok

[  0%]: ccache compiling.release src/main.c
[100%]: linking.release test

If you want to known more, you can see #339

[solvingj]

this is looking very good, amazing progress as usual. it seems you have learned a great deal about Conan, and now have plans on how to improve your support for it in the future. I think continued improvement on this integration will be sufficiently tracked in #339 along with the other package managers in a general way. Overall, there is a mostly-working integration now, so I will close this ticket.

[waruqi]

A simple example

add_requires("CONAN::zlib/1.2.11@conan/stable", {alias = "zlib", debug = true}})
add_requires("CONAN::OpenSSL/1.0.2n@conan/stable", {alias = "openssl", 
    configs = {options = "OpenSSL:shared=True"}})

target("test")
    set_kind("binary")
    add_files("src/*.c") 
    add_packages("openssl", "zlib")

[waruqi]

@solvingj It seems that the new version of conan 2.x breaks a lot of things.
How should I better integrate conan?

/usr/local/bin/conan export /tmp/test/.xmake/linux/x86_64/conan/xmake_generator bincrafters/testing
usage: conan export [-h] [-f FORMAT] [-v [V]] [--name NAME] [--version VERSION] [--user USER] [--channel CHANNEL] [-r REMOTE | -nr] [-l LOCKFILE] [--lockfile-out LOCKFILE_OUT] [--lockfile-partial]
[--build-require]
path
conan export: error: unrecognized arguments: bincrafters/testing

#3633

[solvingj]

Your current quandary is a very difficult consequence of adding integration and support with multiple other tool/ecosystems like Conan, VCPKG, Cmake, and others. Each one is likely to make breaking changes at some point, And it's rarely easy to deal with four "wrapper tools" like Xmake. Now you have to decide how much time and complexity you want to devote to continue supporting the subset of xmake users who also use Conan, and use them together. It's been a few years since initial support was added. Now maybe you have some idea of whether or not a substantial number of people are even doing that. You added a native package manager into xmake, so if I were using Xmake for anything serious, I would probably want to use only xmake packages. Having some dependencies come from one package manager, and others come from another seems completely chaotic and unmanageable.

If you decide to continue support, Conan integration, you probably have two choices:

1. version lock Xmake and Conan, saying that Xmake currentVersion and earlier support only Conan 1x, and after Xmake nextVersion, only Conan 2x is supported. Then update all your code to support Conan 2.
2. Add a global Conan Version "mode" in Xmake which toggles all the behaviors in Xmake to use Conan1 or Conan2 semantics. And then write all the new implementation details you need to support that.

Of note, I only really consider the needs of professional software development teams in my comments. Hobbyists, and open source projects have different wants/needs. I don't know which user base you focus on more with Xmake.

In any case, things are going to get difficult end messy for you and/or any affected users for a while, so I wish you the best of luck.

[waruqi]

Can I integrate xmake into https://github.com/conan-io/conan/tree/release/2.0/conan/tools? Wouldn't that be better?

[solvingj]

Can I integrate xmake into https://github.com/conan-io/conan/tree/release/2.0/conan/tools? Wouldn't that be better?

That folder contains mostly generators for build systems like the xmake-generator we worked on previously, and is still effectively done. There are some other helper functions in there for special cases like CMake, but that's all to help conan invoke and integrate with toolchains and/or build systems which are extremely widespread (cmake, msvc, etc). I don't think they'll want to accept any specific support for xmake into the core codebase, and it's really not necessary. Using the generator from a package is super commonplace. The majority of professional dev teams using Conan have custom generators in packages.

But again, all that is separate from what broke and what you asked about. The topic at hand is the details of how xmake does/should integrate with conan. XMake tries to be a build system, package manager, meta build system and meta package manager. Because it has functionality spanning multiple layers of the vertical tool stack, it's hard to talk about in simple terms. There is a similar quandry with cmake. conan invokes cmake as a build system, when building projects which use cmake as their primary build system. At the same time, many professional teams have decided to make cmake their "golden hammer" / "entrypoint" for all scripting and automation of anything related to building their projects. So, they want to invoke cmake ... and have cmake invoke conan much like you have done with xmake. However, LOTS of challenges exist in that situation.

But getting back to the point, the xmake feature that broke is that xmake is it's meta package management with conan v2. xmake can no longer orchestrate the Conan commands to use it as a external package manager successfully with conan v2 because the CLI structure has changed. The surface problem is simple to fix. Just fix the CLI invocations to format/pass the appropriate args according to the new CLI documentation. But, as I alluded to, you probably want to support both conan v1 and conan v2 CLI and related behaviors. So, you'll need to add a whole new conan v2 support layer and the necessary logic to toggle between them for users. Many users will be stuck on conan v1 client and recipes and packages for a long time, some forever. Many will use conan v2. So, while the surface problem is simple to fix, the long-term maintenance and reliability problem is much more difficult.

[SirLynix]

I may have missed something but can't this be simply solved by querying Conan version before using its CLI, like conan -v?

[solvingj]

I may have missed something but can't this be simply solved by querying Conan version before using its CLI, like conan -v?

Which problem are you suggesting that solves specifically?

If you mean the part about a "Conan v1|v2 mode", sure that could be the default "autodetect" behavior but I would consider that a minor detail. There are a lot of implementation details in the current conan support which will be fundamentally different for Conan v2 so that's a lot of code to re-write/change/maintain. But again, bigger than all that is the strategic decision about whether to support both modes in perpetuity.

[waruqi]

I'm trying to write a new generator for xmake, but I'm having some problems. conan-io/conan#13709

[solvingj]

You're almost there, they'll get you the rest of the way.

[waruqi]

conan 2.0.5 re-supports the export of global generators. conan-io/conan#13718

xmake/dev now also supports conan 2.0.5 #3703

[Jeordanis]

I'm having an issue trying to use the conan support. I'm trying to use a package that is already downloaded in the conan cache: "/Users/user/.conan2". In my xmake.lua file I have:

add_requires("conan::qtmaya/2024", {alias = "qt"})
target("UILibrary")
  set_kind("static")
  add_files("MyWidget.cpp")
  add_packages("qt")

When I run xmake I get:

note: install or modify (m) these packages (pass -y to skip confirm)?
in conan:
-> conan::qtmaya/2024 latest 
please input: y (y/n/m)

Pressing y seems to use the cache. All good. To link the package I can use add_packages("qt") but I actually want to link to specific modules/libraries inside qtmaya/2024

How should I do that? I tried qt::Qt5Core but that doesn't link the library correctly.