conan create unnecessary forces rebuild of build-time dependency

[liberforce]

I'm trying to repackage not only binaries but also libraries.

I was told plugins are what I want, but 1.8 hasn't been released so I have to find other solutions. Mine is to perform some "package inheritance", as I only have minor changes to do. So I'm using python_requires to get a parent's package recipe, and inherit from it in my own recipe. Inheriting allows me to have the same settings and options as the parent, (and thus, the same package id). This is nice, as settings and options can change dynamically in the parent's recipe, so I may follow it without much changes in the child recipe.

Now here's the problem: let's say as an example I'm repackaging zlib. I change the version field to explicit it's a different packaging 'flavor'.

Here's a simplified conanfile.py exposing the problem:

import conans

parent_package = 'zlib/1.2.11@conan/stable'
base = conans.python_requires(parent_package)


class ZlibQiToolchain(base.ZlibConan):
    version = base.ZlibConan.version + '-pkg1'
    build_requires = parent_package
    keep_imports = True

    def source(self):
        pass

    def build(self):
        pass

    def imports(self):
        self.copy('*')

    def package(self):
        self.copy('*')

I change the version field by adding a suffix, so it becomes zlib/1.2.11-pkg1@user/channel, and it has a build-time dependency over zlib/1.2.11@conan/stable.

The problem is the following: each time I try call conan create with this recipe, the build requirements are always rebuilt (most probably because the package name is the same). This is incorrect as although the package names are the same, the versions are not. Only zlib/1.2.11-pkg1@user/channel should be forcedly rebuilt, not zlib/1.2.11@conan/stable.

So I always get this in the logs (Build instead of Cache):

Build requirements packages
    zlib/1.2.11@conan/stable:a79ecbb5554da8510d2c9d7856daeb24de6be0d9 - Build

Using a different package name works, but defeats the purpose: overriding the packages requires them having the same name.

Using Conan 1.7.3 on Ubuntu 14.04.

• I've read the CONTRIBUTING guide.
• I've specified the Conan version, operating system version and any tool that can be relevant.
• I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[danimtb]

This looks like a complex way of reusing everything at the same time. I think the original intention described in #3538 is valid and maybe we should come up with better approach to achieve what you describe.

[danimtb]

Here is a test printing the output you described:

   def reuse_same_name_test(self):
        client = TestClient(servers={"default": TestServer()},
                            users={"default": [("lasote", "mypass")]})
        conanfile = """from conans import ConanFile
class MyConanfileBase(ConanFile):
        name = "my_conanfile_base"
        version = "1.0"
        
        def build(self):
            self.output.info("This this the base build() method")
"""
        client.save({"conanfile.py": conanfile})
        client.run("export . lasote/testing")
        reuse = """from conans import python_requires
parent_package = "my_conanfile_base/1.0@lasote/testing"
base = python_requires(parent_package)

class PkgTest(base.MyConanfileBase):
     version = base.MyConanfileBase.version + '-pkg1'
     build_requires = parent_package
"""
        client.save({"conanfile.py": reuse}, clean_first=True)
        client.run("create . lasote/testing")
        print(client.out)

[jgsogo]

Based on the test provided by @danimtb I have created a branch with a couple of tests more (fin it here: https://github.com/conan-io/conan/compare/develop...jgsogo:issue/3639?expand=1). The last test, test_both fails.

Input files are:

from conans import ConanFile

class BasePck(ConanFile):
        name = "my_conanfile_base"
        version = "1.0"

        def build(self):
            self.output.info(">>> BasePck '{}/{}'".format(self.name, self.version))

from conans import ConanFile, python_requires

requirement = "my_conanfile_base/1.0@lasote/testing"
base = python_requires(requirement)

class BothPck(base.BasePck):
    version = "both"
    build_requires = requirement

    def build(self):
        self.output.info(">>> BothPck '{}/{}'".format(self.name, self.version))

and after creating first package and being sure that it is already in the cache; the output when creating the second one builds again a package that is already in the cache:

Exporting package recipe
my_conanfile_base/both@lasote/testing: A new conanfile.py version was exported
my_conanfile_base/both@lasote/testing: Folder: /private/var/folders/yq/14hmvxm96xd7gfgl37_tnrbh0000gn/T/tmppb2_lmrxconans/path with spaces/.conan/data/my_conanfile_base/both/lasote/testing/export
Configuration:
[settings]
os=Macos
os_build=Macos
arch=x86_64
arch_build=x86_64
compiler=apple-clang
compiler.version=10.0
compiler.libcxx=libc++
build_type=Release
[options]
[build_requires]
[env]

my_conanfile_base/both@lasote/testing: WARN: Forced build from source
my_conanfile_base/1.0@lasote/testing: WARN: Forced build from source
my_conanfile_base/both@lasote/testing: Installing package
Requirements
    my_conanfile_base/both@lasote/testing from local cache - Cache
Python requires
    my_conanfile_base/1.0@lasote/testing
Packages
    my_conanfile_base/both@lasote/testing:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Build
Build requirements
    my_conanfile_base/1.0@lasote/testing from local cache - Cache
Build requirements packages
    my_conanfile_base/1.0@lasote/testing:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Build

my_conanfile_base/1.0@lasote/testing: Building your package in /private/var/folders/yq/14hmvxm96xd7gfgl37_tnrbh0000gn/T/tmppb2_lmrxconans/path with spaces/.conan/data/my_conanfile_base/1.0/lasote/testing/build/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
my_conanfile_base/1.0@lasote/testing: Copying sources to build folder
my_conanfile_base/1.0@lasote/testing: Generator txt created conanbuildinfo.txt
my_conanfile_base/1.0@lasote/testing: Calling build()
my_conanfile_base/1.0@lasote/testing: >>> BasePck 'my_conanfile_base/1.0'
my_conanfile_base/1.0@lasote/testing: Package '5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9' built
my_conanfile_base/1.0@lasote/testing: Build folder /private/var/folders/yq/14hmvxm96xd7gfgl37_tnrbh0000gn/T/tmppb2_lmrxconans/path with spaces/.conan/data/my_conanfile_base/1.0/lasote/testing/build/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
my_conanfile_base/1.0@lasote/testing: Generated conaninfo.txt
my_conanfile_base/1.0@lasote/testing: Generated conanbuildinfo.txt
my_conanfile_base/1.0@lasote/testing: Generating the package
my_conanfile_base/1.0@lasote/testing: Package folder /private/var/folders/yq/14hmvxm96xd7gfgl37_tnrbh0000gn/T/tmppb2_lmrxconans/path with spaces/.conan/data/my_conanfile_base/1.0/lasote/testing/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
my_conanfile_base/1.0@lasote/testing: Calling package()
my_conanfile_base/1.0@lasote/testing: WARN: This conanfile has no package step
my_conanfile_base/1.0@lasote/testing: Package '5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9' created
my_conanfile_base/both@lasote/testing: Applying build-requirement: my_conanfile_base/1.0@lasote/testing
my_conanfile_base/both@lasote/testing: Building your package in /private/var/folders/yq/14hmvxm96xd7gfgl37_tnrbh0000gn/T/tmppb2_lmrxconans/path with spaces/.conan/data/my_conanfile_base/both/lasote/testing/build/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
my_conanfile_base/both@lasote/testing: Configuring sources in /private/var/folders/yq/14hmvxm96xd7gfgl37_tnrbh0000gn/T/tmppb2_lmrxconans/path with spaces/.conan/data/my_conanfile_base/both/lasote/testing/source
my_conanfile_base/both@lasote/testing: Copying sources to build folder
my_conanfile_base/both@lasote/testing: Generator txt created conanbuildinfo.txt
my_conanfile_base/both@lasote/testing: Calling build()
my_conanfile_base/both@lasote/testing: >>> BothPck 'my_conanfile_base/both'
my_conanfile_base/both@lasote/testing: Package '5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9' built
my_conanfile_base/both@lasote/testing: Build folder /private/var/folders/yq/14hmvxm96xd7gfgl37_tnrbh0000gn/T/tmppb2_lmrxconans/path with spaces/.conan/data/my_conanfile_base/both/lasote/testing/build/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
my_conanfile_base/both@lasote/testing: Generated conaninfo.txt
my_conanfile_base/both@lasote/testing: Generated conanbuildinfo.txt
my_conanfile_base/both@lasote/testing: Generating the package
my_conanfile_base/both@lasote/testing: Package folder /private/var/folders/yq/14hmvxm96xd7gfgl37_tnrbh0000gn/T/tmppb2_lmrxconans/path with spaces/.conan/data/my_conanfile_base/both/lasote/testing/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
my_conanfile_base/both@lasote/testing: Calling package()
my_conanfile_base/both@lasote/testing: WARN: This conanfile has no package step
my_conanfile_base/both@lasote/testing: Package '5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9' created

It looks like a bug, not so important (everything ended as expected, although one package is compiled without being necesary), but it looks like hard to debug/fix as it is related with the graph.

[jgsogo]

More on this (another test that fails in my branch): if package/PROJECT and build_requires have the same name, then both are built (even if no --build=pck is given). ⚠️

In case of declaring this dependency through a requires, then an error ERROR: Loop detected is raised ✅

[jgsogo]

Hi! Following the evolution of the graph, after PR #4783 it is impossible to have the same reference with a different version in the same graph, it will be a conflict and one of them should be overridden. So, the situation described here will lead to a loop in the graph, Conan will detect it and fail accordingly.

From our point of view, repackaging using the same name for the reference but changing only the version should be avoided. I'm so sorry if this is an inconvenience, but we need to keep some sanity in the graph and in our minds.

Once again, thank you so much for all the feedback you have been providing to us.