Repackaging a library pulls the wrong binary package

[liberforce]

The context

I'm repackaging the gtest library. Basically, I'm creating a gtest package with a different version, that uses the original gtest as a build dependency. The problem comes from the way the options are interpreted. The final goal is to package an app that depends on my repackaged gtest, and that app has other dependencies, so I need to use -o *:shared=True -o gtest:shared=False because. To simplify, I reproduced the problem just with the recipe below.

The problem

Both packages are named gtest, but gtest:shared=False is applied only to gtest/1.8.0-mypkg@sbre/nightly, and gtest/1.8.0@bincrafters/stable gets *:shared=True. This is wrong, both packages are named gtest, so both should use gtest:shared=False. Because of that, the repackaged verision is marked as being a static lib, but contains a dynamic library, and linking fails on Windows + MSVC 15. Oddly, the CI works on Linux, I need to check how this is possible.

The recipe

import conans

class GTestQiToolchain(conans.ConanFile):
    name = 'gtest'
    version = '1.8.0-mypkg'
    build_requires = 'gtest/1.8.0@bincrafters/stable'
    settings = 'os', 'arch', 'compiler', 'build_type'
    options = {
        'shared': [True, False],
        'build_gmock': [True, False],
        'fPIC': [True, False],
    }
    default_options = {
        'shared': False,
        'build_gmock': True,
        'fPIC': True,
    }
    keep_imports = True

    def configure(self):
        if self.settings.os == 'Windows':
            del self.options.fPIC

    def imports(self):
        self.copy(
            '*',
            root_package=self.name,
            excludes=['conan_imports_manifest.txt', 'conanbuildinfo.txt'],
        )

    def package(self):
        self.copy('*')

The logs:

conan create gtest sbre/nightly --profile ci/conan/profiles/win64-vs2017 -o *:shared=True -o gtest:shared=False --build=missing

Python 2 will soon be deprecated. It is strongly recommended to use Python 3 with Conan:
https://docs.conan.io/en/latest/installation.html#python-2-deprecation-notice

Exporting package recipe
gtest/1.8.0-mypkg@sbre/nightly: A new conanfile.py version was exported
gtest/1.8.0-mypkg@sbre/nightly: Folder: C:\Users\gitlab-runner\.conan\data\gtest\1.8.0-mypkg\sbre\nightly\export
Configuration:
[settings]
os=Windows
os_build=Windows
arch=x86_64
arch_build=x86_64
compiler=Visual Studio
compiler.version=15
build_type=Release
cppstd=14
[options]
*:shared=True
gtest:shared=False
[build_requires]
[env]

gtest/1.8.0@bincrafters/stable: Retrieving from predefined remote 'conan-center'
gtest/1.8.0@bincrafters/stable: Trying with 'conan-center'...
Downloading conanmanifest.txt
[==================================================] 418B/418B
Downloading conanfile.py
[==================================================] 3.9KB/3.9KB
Downloading conan_export.tgz
[==================================================] 758B/758B
Decompressing conan_export.tgz: 766B [00:00, 192kB/s]
Downloading conaninfo.txt
[==================================================] 478B/478B
gtest/1.8.0-mypkg@sbre/nightly: Installing package
Requirements
    gtest/1.8.0-mypkg@sbre/nightly from local cache - Cache
Packages
    gtest/1.8.0-mypkg@sbre/nightly:3f7b6d42d6c995a23d193db1f844ed23ae943226 - Build
Build requirements
    gtest/1.8.0@bincrafters/stable from 'conan-center' - Downloaded
Build requirements packages
    gtest/1.8.0@bincrafters/stable:a35f8fa327837a5f1466eaf165e1b6347f6e1e51 - Download

gtest/1.8.0@bincrafters/stable: Retrieving package a35f8fa327837a5f1466eaf165e1b6347f6e1e51 from remote 'conan-center'
Downloading conanmanifest.txt
[==================================================] 3.6KB/3.6KB
Downloading conaninfo.txt
[==================================================] 478B/478B
Downloading conan_package.tgz
[==================================================] 1.0MB/1.0MB
Decompressing conan_package.tgz: 981kB [00:00, 15.0MB/s]
gtest/1.8.0@bincrafters/stable: Package installed a35f8fa327837a5f1466eaf165e1b6347f6e1e51
gtest/1.8.0-mypkg@sbre/nightly: Applying build-requirement: gtest/1.8.0@bincrafters/stable
gtest/1.8.0-mypkg@sbre/nightly: Building your package in C:\Users\gitlab-runner\.conan\data\gtest\1.8.0-mypkg\sbre\nightly\build\3f7b6d42d6c995a23d193db1f844ed23ae943226
gtest/1.8.0-mypkg@sbre/nightly: Configuring sources in C:\Users\gitlab-runner\.conan\data\gtest\1.8.0-mypkg\sbre\nightly\source
gtest/1.8.0-mypkg@sbre/nightly: Copying sources to build folder
gtest/1.8.0-mypkg@sbre/nightly: Generator txt created conanbuildinfo.txt
gtest/1.8.0-mypkg@sbre/nightly imports(): Copied 41 '.h' files:
gtest/1.8.0-mypkg@sbre/nightly imports(): Copied 2 '.cmake' files: FindGMock.cmake, FindGTest.cmake
gtest/1.8.0-mypkg@sbre/nightly imports(): Copied 2 '.txt' files: conanmanifest.txt, conaninfo.txt
gtest/1.8.0-mypkg@sbre/nightly imports(): Copied 4 '.lib' files: gtest.lib, gmock_main.lib, gmock.lib, gtest_main.lib
gtest/1.8.0-mypkg@sbre/nightly imports(): Copied 1 '.pdb' file: vc141.pdb
gtest/1.8.0-mypkg@sbre/nightly imports(): Copied 4 '.dll' files: gtest.dll, gmock_main.dll, gmock.dll, gtest_main.dll
gtest/1.8.0-mypkg@sbre/nightly: Calling build()
gtest/1.8.0-mypkg@sbre/nightly: WARN: This conanfile has no build step
gtest/1.8.0-mypkg@sbre/nightly: Package '3f7b6d42d6c995a23d193db1f844ed23ae943226' built
gtest/1.8.0-mypkg@sbre/nightly: Build folder C:\Users\gitlab-runner\.conan\data\gtest\1.8.0-mypkg\sbre\nightly\build\3f7b6d42d6c995a23d193db1f844ed23ae943226
gtest/1.8.0-mypkg@sbre/nightly: Generated conaninfo.txt
gtest/1.8.0-mypkg@sbre/nightly: Generated conanbuildinfo.txt
gtest/1.8.0-mypkg@sbre/nightly: Generating the package
gtest/1.8.0-mypkg@sbre/nightly: Package folder C:\Users\gitlab-runner\.conan\data\gtest\1.8.0-mypkg\sbre\nightly\package\3f7b6d42d6c995a23d193db1f844ed23ae943226
gtest/1.8.0-mypkg@sbre/nightly: Calling package()
gtest/1.8.0-mypkg@sbre/nightly package(): Copied 41 '.h' files:
gtest/1.8.0-mypkg@sbre/nightly package(): Copied 2 '.cmake' files: FindGMock.cmake, FindGTest.cmake
gtest/1.8.0-mypkg@sbre/nightly package(): Copied 4 '.txt' files: conanbuildinfo.txt, conaninfo.txt, conanmanifest.txt, conan_imports_manifest.txt
gtest/1.8.0-mypkg@sbre/nightly package(): Copied 4 '.lib' files: gmock.lib, gmock_main.lib, gtest.lib, gtest_main.lib
gtest/1.8.0-mypkg@sbre/nightly package(): Copied 1 '.pdb' file: vc141.pdb
gtest/1.8.0-mypkg@sbre/nightly package(): Copied 4 '.dll' files: gmock.dll, gmock_main.dll, gtest.dll, gtest_main.dll
gtest/1.8.0-mypkg@sbre/nightly: Package '3f7b6d42d6c995a23d193db1f844ed23ae943226' created

Using Conan 1.8.2 on Windows Server 2016 with MSVC 15.

• I've read the CONTRIBUTING guide.
• I've specified the Conan version, operating system version and any tool that can be relevant.
• I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[liberforce]

Works on Linux because I'm using *:shared=True gtest:shared=True there.

[liberforce]

An easy way to see the problem is when the computed binary configuration hash is different:

gtest/1.8.0@bincrafters/stable: Package installed a35f8fa327837a5f1466eaf165e1b6347f6e1e51
gtest/1.8.0-mypkg@sbre/nightly: Package '3f7b6d42d6c995a23d193db1f844ed23ae943226' created

or just seeing that .dll files are ending up in the final (supposedly static) package:

gtest/1.8.0-mypkg@sbre/nightly package(): Copied 4 '.dll' files: gmock.dll, gmock_main.dll, gtest.dll, gtest_main.dll

[danimtb]

Seems related to #3639 as this is a wrong behavior in the propagation of options scoped by library name

[jgsogo]

Yes, it is true that there is a problem when the build_requires has the same name as the package. Not only related to options, also getting the binaries from the cache (here). It is not easy to fix, and probably the best we can do is alert the user (or even raise an error) to prevent an unexpected behavior without noticing.

There is an onging effort to refactor the dependency graph, these issues should be considered there. Please, let us know if this issue is still important to yo in order to spend more or less time trying to solve it before the new graph arrives.

Thanks!

[liberforce]

I switched projects so this is not critical for me. However, the alternative of forking all the recipes of all the software you want to repackages for some simple changes is awful, and plugins don't work for that use case either. It feels like the best solution for us integrators of third party software, as it keeps the code centralized, short, and your packaging changes separate from upstream. You can also either use prisitine upstream recipes, or override on the fly to use your repackaged software, as the package name is the same.

Some of the causes of the repackaging could go away if a package content was more standardized in terms of paths, like in the FHS for example, or what Debian does to have license files consistently go at the same place. But I fear some downstream will always have need for repackaging, for example to remove unwanted files before shipping, and this shouldn't require full rebuilds and hard dependencies on renamed/rebranded packages.

[jgsogo]

The issue with Conan (at the development point where we are now) is that you cannot repackage binaries AND inherit the recipe, but you can do the last one thing, which the main one. I don't find so critical to compile again the binaries for your overriden recipe (in a company you will do it, anyway, you are not going to rely on the binaries generated by someone else).

As I said, we will try to keep this issue live and check it against the new graph model.

Thanks again! 👍

[liberforce]

(in a company you will do it, anyway, you are not going to rely on the binaries generated by someone else).

Well, that's a matter of trust. Did you rebuild your Linux distro from source, or did you trust it to provide binaries on good faith? Now if you say I can't trust the official binaries provided by the Conan team, even though they're signed, then I don't need inheritance at all. Inheritance hides the recipe details, so what I'd need is to fork each package.

However if I can trust Conan, I can avoid forking all the components, and closely follow the packages provided by the community and the fixes in the recipes while keeping our maintenance burden on community-maintained packages as low as possible. I can also report bugs based on the exact stuff you deliver. And then if I trust Conan, repackaging official packages without rebuilding them makes sense.

[jgsogo]

Same as #3639, this kind of repackaging will lead to a loop in the graph: two recipes with the same name requiring each other :/