Conan package issues

[helmesjo]

Just tried to use this instead of my own conan-corrade, and it appears that the cmake scripts in the modules folder aren't properly copied to package. Only file in .conan/data/<package-path>/lib/cmake/Corrade is UseCorrade.cmake.

So when building my conan-magnum with corrade/2019.10 (from conan-center) I get:

CMake Error at source_subfolder/CMakeLists.txt:328 (include):
  include could not find load file:

    C:/.conan/ebef85/1/lib/cmake/Corrade/CorradeLibSuffix.cmake

OS: Windows (can't remember if I've tested on Linux...)

[mosra]

Hi!

I remember a related discussion going about this at https://github.com/conan-io/conan-center-index/pull/283/files#r342450828 but I'm not sure if we concluded it properly before merging (I admit I might have not paid attention at the end, as the discussion was going on quite a long time).

Yes, the CorradeLibSuffix.cmake needs to be installed too, otherwise depending projects won't build. Would it be possible for you to open an issue / PR on the upstream conan repo and reference it here? Also, would the conanfile.py in this project root need to be updated somehow?

Cc: @Croydon

[mosra]

@helmesjo do you know what should be added where to fix this (or, ideally, could you provide a patch)? I looked at the conanfile.py, but I don't see any obvious place where this is done :)

[helmesjo]

Sorry, missed this!

Actually, the recipe seams to be correct, but the prebuilt package (corrade/2019.10, which is on conan-center) is wrong. So I guess it just needs to be removed, rebuilt & uploaded.

I cloned this repo, ran conan create . mosra/testing (branch v2019.10) and referenced that in my conan-magnum recipe by corrade/2019.10@mosra/testing and it built all fine.

@Croydon Maybe you can help with this? :)

[mosra]

The recipe in conan-center has this: https://github.com/conan-io/conan-center-index/blob/0ae829a362c1cc6a20d97e023ca0aafc805797c3/recipes/corrade/all/conanfile.py#L110

tools.rmdir(os.path.join(self.package_folder, 'share'))

so I think that's why, and the bundled conanfile.py here is correct. So if I understand correctly, this has to be fixed in conan-center instead -- @helmesjo can you do that? Be sure to check the original discussion for reasons why this was added there in the first place.

[Croydon]

We can add the CorradeLibSuffix.cmake file to the package and add it to the build folders which is necessary for this.

@helmesjo Could you test it with the magnum package if I edit the Corrade recipe respectively?

I have currently way too much on my plate unfortunately

[Croydon]

@mosra Am I getting it right that the MSVC20XY_COMPATIBILITY flags need to be set while building Corrade AND while consuming it in another project?

[Croydon]

I think I just run into an issue related to mosra/magnum#354

where the UseCorrade.cmake file says that it needs the MSVC2019_COMPATIBILITY flag, but actually looks for CORRADE_MSVC2019_COMPATIBILITY

[mosra]

Ha! You have a new avatar :D

Anyway:

1. The *_COMPATIBILITY flags are autodetected with Corrade's CMake scripts depending on compiler version, so they practically wouldn't even need to be listed/exposed in the conan file.
2. Then, the set of detected / manually set flags is saved into Corrade/configure.h (with a CORRADE_* prefix, so the MSVCXYZW_COMPATIBILITY CMake option turns into CORRADE_MSVCXYZW_COMPATIBILITY), which is generated by CMake from configure.h.cmake.
3. Finally, FindCorrade.cmake parses configure.h and extracts those options, making them available to the user that does find_package(Corrade). This is also what UseCorrade.cmake expects.

mosra/magnum#354 is unrelated to this, it happened only due to an outdated FindCorrade that wasn't aware of the new option.

[helmesjo]

The recipe in conan-center has this: https://github.com/conan-io/conan-center-index/blob/0ae829a362c1cc6a20d97e023ca0aafc805797c3/recipes/corrade/all/conanfile.py#L110

tools.rmdir(os.path.join(self.package_folder, 'share'))

so I think that's why, and the bundled conanfile.py here is correct. So if I understand correctly, this has to be fixed in conan-center instead -- @helmesjo can you do that? Be sure to check the original discussion for reasons why this was added there in the first place.

@mosra
Ah, alright. The kind of unfortunate issues that comes with duplication. 🙃

We can add the CorradeLibSuffix.cmake file to the package and add it to the build folders which is necessary for this.

@helmesjo Could you test it with the magnum package if I edit the Corrade recipe respectively?

I have currently way too much on my plate unfortunately

@Croydon
Sure thing, just let me know when it's changed and I'll test!

[Croydon]

The PR is ready, please have a look here: conan-io/conan-center-index#1335

[helmesjo]

Answered in the PR. All works as expected!

[mosra]

Thank you both 😊 Anything left to be done from my side?

[Croydon]

No, for now it should be fine

Until next time 👋 😄

[helmesjo]

@Croydon Just a quick question. What packages are automatically built for Windows on conan-center?
I'm getting this for my conan-magnum build:

ERROR: Missing binary: corrade/2019.10:db921f651d558029e98affa38764c98af2dc8bb7
corrade/2019.10: WARN: Can't find a 'corrade/2019.10' package for the specified settings, options and dependencies:
- Settings: arch=x86_64, build_type=Release, compiler=Visual Studio, compiler.runtime=MT, compiler.version=15, os=Windows
- Options: build_deprecated=True, shared=False, with_interconnect=True, with_main=True, with_pluginmanager=True, with_testsuite=True, with_utility=True

[Croydon]

https://github.com/conan-io/conan-center-index/wiki/Supported-Platforms-And-Configurations

Have you set a non-default option value?

[helmesjo]

Not from the looks of it, no... It seems to be the same as I'm requesting:

Mine:

- Settings: arch=x86_64, build_type=Release, compiler=Visual Studio, compiler.runtime=MT, compiler.version=15, os=Windows
- Options: 
build_deprecated=True, 
shared=False, 
with_interconnect=True, 
with_main=True, 
with_pluginmanager=True, 
with_testsuite=True, 
with_utility=True

Corrade defaults:

default_options = {
        "shared": False,
        "fPIC": True,
        "build_deprecated": True,
        "with_interconnect": True,
        "with_main": True,
        "with_pluginmanager": True,
        "with_testsuite": True,
        "with_utility": True,
    }

[Croydon]

You are right. For some odd reasons we only have Linux builds right now and most of them are outdated. I will report this

[Croydon]

conan-io/conan-center-index#1397

[helmesjo]

1. The *_COMPATIBILITY flags are autodetected with Corrade's CMake scripts depending on compiler version, so they practically wouldn't even need to be listed/exposed in the conan file.
   [...]

@mosra Is above actually true, that these are detected (and set?) automatically? While waiting for prebuilt binaries I change my windows builds to build missing dependencies, and get the following error (and equivalent for VS2017) when it builds corrade:

CMake Error at C:/.conan/e89479/1/lib/cmake/Corrade/UseCorrade.cmake:59 (message):
  To use Corrade with MSVC 2019, build it with MSVC2019_COMPATIBILITY enabled

Could they be set automatically? Or should something be changed in the corrade recipe?

Edit
Oh right, that was part of what we discussed above, and is actually set here. 🙃 Odd I get this issue now...

[IAmNotHanni]

I have the same problem
To use Corrade with MSVC 2019, build it with MSVC2019_COMPATIBILITY enabled

[mosra]

Honestly, why is Conan such an endless source of trouble all the time? I don't remember any other packaging system requiring such an extreme maintenance effort, even vcpkg mostly just works, and over the years all other packages combined required maybe 10% of effort that has been put into Conan so far. All these issues are only because Conan replaces/reinvents/rewrites the buildsystem that "just worked" for nearly a decade. I'm getting tired of spending extra time repeatedly explaining and defending all buildsystem decisions I ever did only to have them replicated in an incomplete way in Conan.

</rant>

Is above actually true

I have a quite extreme CI setup plus a lot of day-to-day users on MSVC that would be affected. I don't remember any issues (other than quickly resolved user errors) related to this since 2015, so I'd say yes it's true.

Pretty sure I already explained this in one of the 100-message threads about Conan, but let's go over it again once more:

1. Corrade's buildsystem detects the compiler, grabs all options used at configure time and based on that generates Corrade/configure.h that's then installed along with other includes. That file then has #define CORRADE_MSVC2019_COMPATIBILITY if the option is enabled, and similarly for other compilers / versions as well as stuff like CORRADE_BUILD_MULTITHREADED, CORRADE_BUILD_STATIC etc., which are also important but don't immediately break the build like this one.
2. When finding the package, FindCorrade.cmake parses the configure.h and exposes the defines as CMake variables, so when you find_package(Corrade) you have the same CMake variables defined as you have in a C++ preprocessor. These are then picked up by UseCorrade.cmake to do various checks, decide about compiler flags and many other things.

Because Conan threw away all that's in FindCorrade.cmake and remade (a very small subset of it) from scratch, the part that parses configure.h is simply missing. So, in order to fix this, you need to do the following in Conan as well:

corrade/modules/FindCorrade.cmake

Lines 300 to 331 in a119b35

	# Read flags from configuration
	file(READ ${_CORRADE_CONFIGURE_FILE} _corradeConfigure)
	string(REGEX REPLACE ";" "\\\\;" _corradeConfigure "${_corradeConfigure}")
	string(REGEX REPLACE "\n" ";" _corradeConfigure "${_corradeConfigure}")
	set(_corradeFlags
	MSVC2015_COMPATIBILITY
	MSVC2017_COMPATIBILITY
	MSVC2019_COMPATIBILITY
	BUILD_DEPRECATED
	BUILD_STATIC
	BUILD_MULTITHREADED
	TARGET_UNIX
	TARGET_APPLE
	TARGET_IOS
	TARGET_IOS_SIMULATOR
	TARGET_WINDOWS
	TARGET_WINDOWS_RT
	TARGET_EMSCRIPTEN
	TARGET_ANDROID
	# TARGET_X86 etc and TARGET_LIBCXX are not exposed to CMake as the meaning
	# is unclear on platforms with multi-arch binaries or when mixing different
	# STL implementations. TARGET_GCC etc are figured out via UseCorrade.cmake,
	# as the compiler can be different when compiling the lib & when using it.
	PLUGINMANAGER_NO_DYNAMIC_PLUGIN_SUPPORT
	TESTSUITE_TARGET_XCTEST
	UTILITY_USE_ANSI_COLORS)
	foreach(_corradeFlag ${_corradeFlags})
	list(FIND _corradeConfigure "#define CORRADE_${_corradeFlag}" _corrade_${_corradeFlag})
	if(NOT _corrade_${_corradeFlag} EQUAL -1)
	set(CORRADE_${_corradeFlag} 1)
	endif()
	endforeach()

The same needs to be done for Magnum, where the set of such flags is even larger.

[mosra]

@Croydon see above. These flags are not meant to be passed to CMake other than when building Corrade itself, when using it the flags are meant to be extracted from Corrade/configure.h and exposed to CMake, not supplied externally. That's in order to avoid accidents with different set of flags being used for building and using the library.

There being a prefixed and unprefixed version is orthogonal to this and while specifying the prefixed version also seemingly makes this issue go away, it's not a proper fix.

[helmesjo]

@Croydon
Release 2020.06 is out. How does one get the new build into conan-center (I keep forgetting these details)? :)

[helmesjo]

I made a PR.

Built it locally and modified conan-magnum to reference the locally built package and that also built correctly, and test_package passed (had to add NOMINMAX define to CMakeLists.txt script in conan-magnum).

[helmesjo]

Note regarding the To use Corrade with MSVC 2019, build it with MSVC2019_COMPATIBILITY enabled issues (for all MSVC versions):

I got this again (in my dependent project) after updating magnum to 2020.06 (local build), and overall it seams to be a recurring issue. BUT, it works (my project which uses a bunch of Magnum stuff) if I remove this line in the corrade recipe (that is, not packaging UseCorrade.cmake for consumers). I can't remember why we added this (do you know @Croydon?), the one missing (that started this thread) was CorradeLibSuffix.cmake.

SO, correct me if I'm wrong @mosra but I understand that UseCorrade.cmake is only needed during build and shouldn't be passed on to consumers (pretty sure you've stressed this a few times but want to be sure 😇). This is also what my recent digging shows. So let's remove remove this line. If there are no objections I'll include it in my PR.

[mosra]

I got this again

Yes, again, this needs to be fixed by extracting the flags out of Corrade/configure.h, which is what FindCorrade.cmake does and which is what you need to replicate in Conan, as I said above.

understand that UseCorrade.cmake is only needed during build

No, you got it backwards -- UseCorrade is full of functions and macros that are useful for CMake consumers (test setup, plugin compilation, useful compiler flags etc). For non-CMake consumers it has no use.

[helmesjo]

Alright, so after a few hours spent here and there, I hopefully got it figured out. The corrade recipe now exposes the UseCorrade.cmake/FindCorrade.cmake (etc) scripts for consumers so that configuration.h is read correctly.

It also correctly sets the CMake defines used by corrade, including the "Visual Studio Compatibility" ones.

Also fixed rpath stuff for dynamic linking.

I've tested it all with my magnum conan package, and it works as expected AFAIK.

[pezcode]

Isn't the point of Conan to make the package independent of the build system used to compile it? CCI has an FAQ entry on the topic and will most likely error out on the hook.

[helmesjo]

Alright, missed that one. Then I guess (regarding that part) we are back to what @mosra mentioned that the recipe must support parsing of configuration.h and feed that to consumers. Shouldn't be too much trouble, but unfortunately a bit error prone if (when) something changes in Corrade.

[Croydon]

Isn't the point of Conan to make the package independent of the build system used to compile it

In general yes, but if projects like Corrade define and ship custom CMake functions which go beyond more universal and standardized things like defining targets and finding libraries, then the only thing we can do is exposing this for CMake consumers. You can still use the Conan package with other build systems, but you will naturally not be able to use Corrade's CMake functions. if this is still useful depends and is another question.

[helmesjo]

@Croydon Yes that was my take on it, and as @mosra mentioned these CMake scripts contain stuff that's (obviously) only relevant for CMake users. So basically the idea is to hijack this, put it in a ConfigCorrade.cmake script and package as a build module.
Am I correct that this will then only be used by the cmake generator(s), and won't affect other build systems?

[Croydon]

Am I correct that this will then only be used by the cmake generator(s), and won't affect other build systems?

Correct

[Janos95]

@helmesjo what about cmake functions like corrade_add_resource, could those also be exported to the CorradeConfig.cmake? Also on another note, I noticed that conan now supports components. It would be very cool to use this feature in this recipe in order to be able to explicitly link against the Corrade::Containers, Corrade::Utilities, etc targets.

[helmesjo]

@Janos95 Yep, that as well... So my second take on this is that we want to setup everything the way the Corrade cmake scripts already do except anything regarding target(s), linking etc. And that will pass the hook, correct? @Croydon

[Janos95]

Regarding the appropriate target names. My understanding is that in order to be able to e.g. consume the Corrade::Containers target one has to be more explicit in the cpp_info. I think the recipe needs to contain something along the lines of

deps = {
  "Containers": ["Utilities"],
  "Utilities": ["Containers"],
  "Interconnect": ["Containers"],
  "PluginManager": ["Containers"],
  "TestSuite": ["Containers"],
  "Main": []
}
for component in ["Containers", "Utilities", "Interconnect", "PluginManager", "TestSuite", "Main"]:
    lib = "Corrade" + component + suffix
    if lib in builtLibs:
        self.cpp_info.components[component].names["cmake_find_package"] = component
        self.cpp_info.components[component].names["cmake_find_package_multi"] = component
        self.cpp_info.components[component].libs = [lib]
        self.cpp_info.components[component].requires = deps[component]

[Croydon]

So my second take on this is that we want to setup everything the way the Corrade cmake scripts already do except anything regarding target(s), linking etc. And that will pass the hook, correct?

Yes. And to get granular targets @Janos95 point our correctly that adding Conan components is the way to go.

However, you might want to split this work to not fight too many battles at once

[helmesjo]

However, you might want to split this work to not fight too many battles at once

Yeah, I'll get the configuration stuff working first and foremost. 🙃

[helmesjo]

Not seeing any issues atm with the conan-center package, so I'll close this for the time being.