Missing prebuilt package

[db4]

I have a binary package uploaded to a local Artifactory 6.9.1 repo, but conan fails to download it:

dbely@bely-win81:~$ conan --version
Conan version 1.15.0
dbely@bely-win81:~$ conan search -r conan-local TBB/4.4.4@conan/stable -q 'arch=x86_64 AND build_type=Release AND compiler=gcc AND compiler.libcxx=libstdc++11 AND compiler.version=5 AND os=Linux'
Existing packages for recipe TBB/4.4.4@conan/stable:

Existing recipe in remote 'conan-local':

    Package_ID: aa0498c70db8439f7f74687e9366567b85010fa5
        [options]
            shared: True
        [settings]
            arch: x86_64
            build_type: Release
            compiler: gcc
            compiler.libcxx: libstdc++11
            compiler.version: 5
            os: Linux
        Outdated from recipe: False

dbely@bely-win81:~$ conan install TBB/4.4.4@conan/stable -o TBB:shared=True
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=5
os=Linux
os_build=Linux
[options]
TBB:shared=True
[build_requires]
[env]

Installing package: TBB/4.4.4@conan/stable
Requirements
    TBB/4.4.4@conan/stable from 'conan-center' - Cache
Packages
    TBB/4.4.4@conan/stable:aa0498c70db8439f7f74687e9366567b85010fa5 - Missing

TBB/4.4.4@conan/stable: WARN: Can't find a 'TBB/4.4.4@conan/stable' package for the specified settings, options and dependencies:
- Settings: arch=x86_64, build_type=Release, compiler=gcc, compiler.libcxx=libstdc++11, compiler.version=5, os=Linux
- Options: shared=True
- Dependencies:
- Package ID: aa0498c70db8439f7f74687e9366567b85010fa5

ERROR: Missing prebuilt package for 'TBB/4.4.4@conan/stable'
Try to build it from sources with "--build TBB"
Or read "http://docs.conan.io/en/latest/faq/troubleshooting.html#error-missing-prebuilt-package"

dbely@bely-win81:~$

Why?

[memsharded]

Hi @db4

I am having a look to this. What happens if you try the following?

$ conan install TBB/4.4.4@conan/stable -o TBB:shared=True -r=conan-local

It might help to understand the issue the output of:

$ conan remote list_ref

[db4]

conan install TBB/4.4.4@conan/stable -o TBB:shared=True -r=conan-local

That works, but I actually use it in another recipe so wouldn't like to install explicitly...

$ conan remote list_ref

leptonica/1.76.0@bincrafters/stable: conan-center
libjpeg/9b@bincrafters/stable: conan-center
cmake_config_tools/0.0.2@dbely/testing: dbely
libpng/1.6.34@bincrafters/stable: conan-center
jasper/2.0.14@conan/stable: conan-center
libtiff/4.0.9@bincrafters/stable: conan-center
tesseract/4.0.0@bincrafters/stable: conan-center
giflib/5.1.4@bincrafters/stable: conan-center
zlib/1.2.11@conan/stable: conan-center
libtiff/4.0.8@bincrafters/stable: conan-center
TBB/4.4.4@conan/stable: conan-center
libwebp/0.6.1@bincrafters/stable: conan-center
cmake_installer/3.11.3@conan/stable: conan-center
libjpeg/9c@bincrafters/stable: conan-center

I thought it was allowed for some time to upload a missing binary package to another remote. For other recipes it works but surprisingly not for TBB...

[danimtb]

Did you manage to reproduce the issue @db4?

[db4]

Did you manage to reproduce the issue @db4?

Of course, it happens all the time. Nothing has changed since I reported it. But maybe conan just works as expected? Should it search other remotes for a binary package?

[danimtb]

Does it happen only with the TBB package as you reported or with all of them? I think we have tests covering this case, but not sure what is the default behavior right now

[db4]

@danimtb

Does it happen only with the TBB package as you reported or with all of them?

Looks like that with all of them. I just see the same behaviour with cygwin_installer

[db4]

@memsharded @danimtb any update on the issue? Looks like the current behavior (never search other remotes for binaries) was introduced in #3890. Frankly speaking, I don't quite understand the purpose.

[danimtb]

Hi @db4 and thanks for pointing that issue.

I have looked in the Conan test suite and it seems that you case is covered by this test https://github.com/conan-io/conan/blob/develop/conans/test/integration/multi_remote_test.py#L125

It has been in place since Conan 1.9.1 and the behavior has not changed since then.

The explanation behind this behavior is that normally packages are generated with a specific recipe, so only the packages uploaded with the recipe are supposed to be reproducible with that recipe. However, the same reference (TBB/4.4.4@conan/stable in your case) uploaded to another remote could not be the same recipe. Mixing packages created with other recipe could lead to inconsistencies in the reproducibility of the packages.

We decided to keep the behavior consistent with recipes and its associated packages and force the mix of packages explicitly with conan install <> --remote <remote>

Thanks for reporting. I think this information should be added to our FAQ, as it something important.

Hope it makes sense

[db4]

@danimtb

I have looked in the Conan test suite and it seems that you case is covered by this test https://github.com/conan-io/conan/blob/develop/conans/test/integration/multi_remote_test.py#L125

So what conditions should be satisfied to make conan search multiple remotes for a binary package?

The explanation behind this behavior is that normally packages are generated with a specific recipe, so only the packages uploaded with the recipe are supposed to be reproducible with that recipe. However, the same reference (TBB/4.4.4@conan/stable in your case) uploaded to another remote could not be the same recipe. Mixing packages created with other recipe could lead to inconsistencies in the reproducibility of the packages.

Isn't package ID affects binary package ID? I always thought it should and so if the recipe changes all binary package IDs change as well.

Even it's not the case, why conan cannot test if a remote contains the same recipe (by ID) and only then search binaries there?

Returning to my specific problem. In addition to exising binaries on conan-center I need some additional ones (missing there) and I don't like to build them once again every time CI job is run. In the past I just used conan-package-tools/CONAN_UPLOAD_DEPENDENCIES=all (or did that manually via conan upload), and missing binaries had built only once. How to achieve that nowadays?

[danimtb]

The package ID is calculated from the recipe settings, options... but a change in the recipe does not imply a change in the package ID.

The ID of the recipe is the reference you give to them. Conan is not checking the contents of the recipe to see if it is the same inside. There are some cases where you would like that behavior and other ones that don't.

The solution to your issue would be the revisions feature, where the recipe is associated with an automatic version created from its contents or from a git commit.

Having the revision of the recipe makes Conan know if a recipe is the same in different remotes (additionally to the reference), and in that scenario mixing binaries from different remotes is done by default. However, take into account that the revisions have different implications and would change some of the flows.

[db4]

The package ID is calculated from the recipe settings, options... but a change in the recipe does not imply a change in the package ID.

OK, but any remote anyway holds the recipe's MD5 hash. Why conan client cannot search for binaries only remotes with the identical recipe's hash value?

I understand it requires some work but I think that "safe" pre 1.9.1 behavior would be very convenient...

The solution to your issue would be the revisions feature, where the recipe is associated with an automatic version created from its contents or from a git commit.

Probably, but switching to revisions seems to be a hard move for me, at least for now.

[danimtb]

Changing the way Conan search for binaries in remote will be a breaking behavior. We considered the approach of mixing binaries but concluded that the assumption of users wanting that behavior might be confusing and hard to explain when trying to achieve consistent results in the binary graph computation.

We strongly believe that the way to go is revisions, which cover not only this case but other relevant issues like reproducibility, automatic versioning of the packages, graph locking... The implementation to achieve this behavior is following another path, and making the hash check will be an extra effort being sure that that is not a solution in the right direction.

[memsharded]

This tickets seems very outdated. Conan 1.X added revisions, and they are by default in 2.0.
Closing this as outdated, thanks.