[question] 2.0 Tool Requires

[beenjohn]

When generating a lockfile in 2.0, the tool requires don't end up in the lock file. With cmake listed as a tool requires, "conan create ... --lockfile=..." return an error that cmake isn't listed in the lockfile. Is this expected behavior?

• I've read the CONTRIBUTING guide.

[czoido]

Hi @beenjohn,

Can you provide more details, maybe a reproducible example?

For me, with a dummy recipe like this:

from conan import ConanFile
from conan.tools.cmake import cmake_layout


class mylibRecipe(ConanFile):
    name = "mylib"
    version = "1.0"
    # Binary configuration
    settings = "os", "compiler", "build_type", "arch"

    def layout(self):
        cmake_layout(self)

    def requirements(self):
        self.requires("zlib/1.2.11")
        self.tool_requires("cmake/3.19.8")

And doing:

conan lock create .
conan create . --lockfile=conan.lock

Works, creating a lockfile like this one:

{
    "version": "0.5",
    "requires": [
        "zlib/1.2.11#4524fcdd41f33e8df88ece6e755a5dcc%1650538915.154"
    ],
    "build_requires": [
        "cmake/3.19.8#7c53e8476ea7485f06e2b463d88f6787%1652284298.374"
    ],
    "python_requires": []
}

[beenjohn]

@czoido,

Apologies for the delayed response. To clarify, the issue I'm seeing in 2.0.0-beta2 is that the tool_requires in the test_package is not ending up in the lockfile, but conan create fails due to the missing requirement. See below:

libsodium (rough draft) 2.X conanfile:

from conan import ConanFile
from conan.tools import files as conan_files
from conan.tools.microsoft import msvc_runtime_flag
from conan.tools.apple.apple import is_apple_os
from conans.util.env import get_env
from conan.tools.gnu import Autotools
from conan.tools.gnu import AutotoolsToolchain
from conans.errors import ConanInvalidConfiguration
import os

required_conan_version = "2.0.0-beta2"

class LibsodiumConan(ConanFile):
    name = "libsodium"
    version = "1.0.18"

    description = "A modern and easy-to-use crypto library."
    license = "ISC"
    url = "https://github.com/conan-io/conan-center-index"
    homepage = "https://doc.libsodium.org/"
    topics = ("sodium", "libsodium", "encryption", "signature", "hashing")

    settings = "os", "arch", "compiler", "build_type"
    options = {
        "shared": [True, False],
        "fPIC": [True, False],
        "use_soname": [True, False],
        "PIE": [True, False],
    }
    default_options = {
        "shared": False,
        "fPIC": True,
        "use_soname": True,
        "PIE": False,
    }

    exports_sources = "patches/*"

    short_paths = True

    _autotools = None

    @property
    def _settings_build(self):
        return getattr(self, "settings_build", self.settings)

    @property
    def _is_msvc(self):
        return str(self.settings.compiler) in ["Visual Studio", "msvc"]

    @property
    def _is_mingw(self):
        return self.settings.os == "Windows" and self.settings.compiler == "gcc"

    def config_options(self):
        if self.settings.os == "Windows":
            del self.options.fPIC

    def configure(self):
        if self.options.shared:
            del self.options.fPIC
        del self.settings.compiler.libcxx
        del self.settings.compiler.cppstd

    def validate(self):
        if self.info.options.shared and self._is_msvc and "MT" in msvc_runtime_flag(self):
            raise ConanInvalidConfiguration("Cannot build shared libsodium libraries with static runtime")

    def generate(self):
        tc = AutotoolsToolchain(self)
        tc.generate()

    def build_requirements(self):
        if not self._is_msvc:
            if self._is_mingw:
                self.build_requires("libtool/2.4.6")
            if self._settings_build.os == "Windows" and not get_env("CONAN_BASH_PATH"):
                self.build_requires("msys2/cci.latest")

    def source(self):
        conan_files.get(self, **self.conan_data["sources"][self.version], strip_root=True)

    @property
    def _msvc_sln_folder(self):
        if self.settings.compiler == "Visual Studio":
            folder = {
                "10": "vs2010",
                "11": "vs2012",
                "12": "vs2013",
                "14": "vs2015",
                "15": "vs2017",
                "16": "vs2019",
                "17": "vs2022",
            }
        else:
            folder = {
                "190": "vs2015",
                "191": "vs2017",
                "192": "vs2019",
                "193": "vs2022",
            }
        return folder.get(str(self.settings.compiler.version))

    def _build_msvc(self):
        msvc_sln_folder = self._msvc_sln_folder or "vs2022"
        upgrade_project = self._msvc_sln_folder is None
        sln_path = os.path.join(self.build_folder, "builds", "msvc", msvc_sln_folder, "libsodium.sln")
        build_type = "{}{}".format(
            "Dyn" if self.options.shared else "Static",
            "Debug" if self.settings.build_type == "Debug" else "Release",
        )
        msbuild = MSBuild(self)
        msbuild.build(sln_path, upgrade_project=upgrade_project, platforms={"x86": "Win32"}, build_type=build_type)

    def _configure_autotools(self):
        if self._autotools:
            return self._autotools
        self._autotools = Autotools(self)
        if self._is_mingw:
            self._autotools.libs.append("ssp")

        if self.settings.os == "Emscripten":
            self.output.warn("os=Emscripten is not tested/supported by this recipe")
            # FIXME: ./dist-build/emscripten.sh does not respect options of this recipe

        yes_no = lambda v: "yes" if v else "no"
        args = [
            "--enable-shared={}".format(yes_no(self.options.shared)),
            "--enable-static={}".format(yes_no(not self.options.shared)),
            "--enable-soname-versions={}".format(yes_no(self.options.use_soname)),
            "--enable-pie={}".format(yes_no(self.options.PIE)),
        ]

        self._autotools.configure(args=args)
        return self._autotools

    def build(self):
        for patch in self.conan_data.get("patches", {}).get(self.version, []):
            conan_files.patch(self, **patch)
        if self._is_msvc:
            self._build_msvc()
        else:
            if self._is_mingw:
                self.run("{} -fiv".format(get_env("AUTORECONF")), cwd=self.source_folder)
            if is_apple_os(self.settings.os):
                # Relocatable shared lib for Apple platforms
                conan_files.replace_in_file(
                    self,
                    "configure",
                    "-install_name \\$rpath/",
                    "-install_name @rpath/"
                )
            autotools = self._configure_autotools()
            autotools.make()

    def package(self):
        conan_files.copy(self, "*LICENSE", src=self.source_folder, dst="licenses", keep_path=False)
        if self._is_msvc:
            conan_files.copy(self, "*.lib", src=self.source_folder, dst="lib", keep_path=False)
            conan_files.copy(self, "*.dll", src=self.source_folder, dst="bin", keep_path=False)
            inc_src = os.path.join(self.source_folder, "src", self.name, "include")
            conan_files.copy(self, "*.h", src=inc_src, dst="include", keep_path=True, excludes=("*/private/*"))
        else:
            autotools = self._configure_autotools()
            autotools.install()
            conan_files.rmdir(self, os.path.join(self.package_folder, "lib", "pkgconfig"))
            conan_files.rm(self, "*.la", os.path.join(self.package_folder, "lib"))

    def package_info(self):
        self.cpp_info.set_property("pkg_config_name", "libsodium")
        self.cpp_info.libs = ["{}sodium".format("lib" if self._is_msvc else "")]
        if not self.options.shared:
            self.cpp_info.defines = ["SODIUM_STATIC"]
        if self.settings.os in ("FreeBSD", "Linux"):
            self.cpp_info.system_libs.append("pthread")
        if self._is_mingw:
            self.cpp_info.system_libs.append("ssp")

libsodium test_package conanfile:

from conan import ConanFile
from conan.tools.cmake import CMake
from conan.tools.cmake import CMakeToolchain
from conan.tools.cmake import CMakeDeps
from conan.tools.build import cross_building
import os


class TestPackageConan(ConanFile):
    settings = "os", "compiler", "arch", "build_type"

    def generate(self):
        tc = CMakeToolchain(self)
        tc.generate()
        deps = CMakeDeps(self)
        deps.generate()

    def build_requirements(self):
        self.tool_requires("cmake/[>=3.21.4]")

    def requirements(self):
        self.requires(self.tested_reference_str)

    def build(self):
        cmake = CMake(self)
        cmake.configure()
        cmake.build()

    def test(self):
        if not cross_building(self):
            self.run("./test_package", env="conanrun")

Lock creation:

conan lock create . && cat conan.lock
{
    "version": "0.5",
    "requires": [],
    "build_requires": [],
    "python_requires": []
}
```

Package build:
```
conan create --lockfile=conan.lock .
Using lockfile: '.../conan.lock'
Exporting the recipe
libsodium/1.0.18: Exporting package recipe
libsodium/1.0.18 exports: File 'conandata.yml' found. Exporting it...
libsodium/1.0.18: Copied 1 '.yml' file: conandata.yml
libsodium/1.0.18 exports: Copied 1 '.yml' file: conandata.yml
...
-------- Computing dependency graph --------
ERROR: Requirement 'cmake/[>=3.21.4]' not in lockfile
```

[czoido]

Hi @beenjohn,
Thanks for the details on how to reproduce the issue, now I can reproduce the problem.
I'm going to mark this as a look-into so I can discuss with the team. Although the fail makes sense because the lockfile is created for the package and it's dependencies and the test_package is not a dependency but a consumer of the package failing there can be confusing.
I'm also adding a failing test to a PR to test this.

Thanks!

[czoido]

@beenjohn,

In the meantime, please check these comments: #11841 (comment)

[beenjohn]

@czoido,

I agree that the failure makes sense given the lockfile is for the package rather than the test package. Being new to the community I wasn't quite sure of the "conan-approved" way to handle this type of setup, but I have a workaround for the time being. Thank you for digging into this in such a timely manner!

I'll leave the ticket open for your tracking, but feel free to close it whenever you like.

[memsharded]

With this test #11841 we are capturing the behavior and the flow that allows to capture the test_package extra requirements in a lockfile, and also how a --lockfile-partial can be used to relax the need for it. We have considered a couple of UX improvements, but nothing clear so far, so recommending this flow atm, and we will keep learning from further feedback. Closing ticket, thanks for the feedback!