run_exports not applied when channel does not provide channeldata.json

[jjhelmus]

As of #3561, run exports are extracted from channeldata.json rather than the packages themselves. This speeds up rendering time but will cause run exports to not be applied if the channel from which the package is supplied does not provide channeldata.json. Most channels on Anaconda.org do not provide channeldata.json.

A simple example:

package:
  name: exporty
  version: 1.0.0

build:
  run_exports:
    strong:
      - exporty

package:
  name: example
  version: 1.0.0

requirements:
  build:
    - exporty

Building exporty and locally and rendering example gives a correct rendering with run_export applied:

$ conda render example/
--------------
Hash contents:
--------------
{}
----------
meta.yaml:
----------
package:
    name: example
    version: 1.0.0
requirements:
    build:
        - exporty 1.0.0 0
    run:
        - exporty
extra:
    copy_test_source_files: true
    final: true

Using the uploaded exporty package on Anaconda.org results in no run_exports applied:

$ conda render -c jjhelmus example/
--------------
Hash contents:
--------------
{}
----------
meta.yaml:
----------
package:
    name: example
    version: 1.0.0
requirements:
    build:
        - exporty 1.0.0 0
extra:
    copy_test_source_files: true
    final: true

[mariusvniekerk]

@jjhelmus is this why we are missing run_exports for our conda-forge aarch64 and ppc64le packages?

[jjhelmus]

@jjhelmus is this why we are missing run_exports for our conda-forge aarch64 and ppc64le packages?

Yes, moving the effected conda-build aarch64 packages (3.18.6 3.18.5 3.18.4) to a broken label may be the easiest fix until this is changed in conda-build itself.

[minrk]

There's something wrong with run_exports on linux-64 as well. When I run conda render on a package that has fftw as a host dependency, it's getting run_exports coming from builds of packages not in the host environment. Seems like something's amiss in general in the run_exports computation.

meta.yaml:

$ cat test/meta.yaml
package:
  name: testing
  version: 1.0

source:

build:
  number: 1

requirements:
  build: []
  host:
    - fftw
  run:
    - fftw

conda render --verbose test

Adding in variants from internal_defaults
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
Attempting to finalize metadata for testing
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
--------------
Hash contents:
--------------
{}
----------
meta.yaml:
----------
package:
    name: testing
    version: '1.0'
build:
    number: '1'
requirements:
    host:
        - libgcc-ng 9.1.0 hdf63c60_0
        - libgfortran-ng 7.3.0 hdf63c60_0
        - fftw 3.3.8 nompi_h7f3a6c3_1105
    run:
        - fftw * mpi_openmpi_*
extra:
    copy_test_source_files: true
    final: true

conda info

     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /home/conda/.condarc
 populated config files : /home/conda/.condarc
          conda version : 4.7.5
    conda-build version : 3.18.6
         python version : 3.7.3.final.0
       virtual packages :
       base environment : /opt/conda  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /home/conda/.conda/pkgs
       envs directories : /opt/conda/envs
                          /home/conda/.conda/envs
               platform : linux-64
             user-agent : conda/4.7.5 requests/2.22.0 CPython/3.7.3 Linux/4.9.125-linuxkit centos/6.10 glibc/2.12
                UID:GID : 9001:9001
             netrc file : None
           offline mode : False

FWIW, I confirmed that this does not happen with conda-build 3.18.3 and does with 3.18.4-6.

[minrk]

For further detail about what might make this case weird: there are three builds of fftw 3.3.8: nompi_, which has no run_exports, and 2x mpi_{provider} which have run_exports: fftw * mpi_{provider}_*. I'm guessing the new logic in #3561 is merging them somehow, or clobbering the one empty run_exports with one of the others.

[SylvainCorlay]

Sadly, we released over 20 packages on the xeus stack making use of the broken version of conda build unfortunately, and there are as many binders that are broken.

[msarahan]

Maybe it would be worth your time to add tests to the conda forge infrastructure or whatever you use so that this doesn't bite you again. Or perhaps a test in conda build's test suite?

…

On Wed, Jul 3, 2019, 12:19 Sylvain Corlay ***@***.***> wrote: Sadly, we released over 20 packages on the xeus stack making use of the broken version of conda build unfortunately, and there are as many binders that are broken. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3583?email_source=notifications&email_token=AAAJL6MDBOUNJUGRMI6K7PLP5TNRVA5CNFSM4H4IVTPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZFD7HQ#issuecomment-508182430>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAJL6NXA4XAIVFZLNVARSDP5TNRVANCNFSM4H4IVTPA> .

[SylvainCorlay]

We test that artefacts are properly produced. Although run-exports being completely ignored looks like a rather major issue for a release...

[msarahan]

But you apparently don't test that libraries are correctly loaded at test time - if you did, you would have caught this.

Run-exports is not completely ignored. It works fine for channels that have channeldata.json. It is tested, and those tests pass. Since you depend on a channel that does not have channeldata.json, perhaps it is worth your time to invest time in conda-build so that it won't break for you in the future.

[SylvainCorlay]

Yeah we are not testing the produced runtime but that should be easily fixed (trying to launch the kernel at least if not running the test suite).

Investing into conda-build core at the moment would really hard in terms of bandwidth for the team.

[msarahan]

I have never asked you to fix it. I have asked you to help write tests for functionality that you depend on, especially when our tests cover our user cases, but not yours.

[SylvainCorlay]

I have asked you to help write tests for functionality that you depend on.

Sounds good. Would be happy to add such tests!

[minrk]

Testing issues like this would be helped a lot by conda-build supporting multiple test envs. For instance, this issue snuck through on mpich because the test environment had the compiler packages, which included the dependencies missing from run_exports. It would also enable testing e.g. ABI compatibility by running tests against latest and oldest allowed versions of dependencies, which are currently not testable.

[minrk]

FWIW, this does not appear to be fixed in 3.18.7. this aarch64 build is still missing run_export s from gfortran

See in the tests for openmpi:

The following NEW packages will be INSTALLED:

    mpi:     1.0-openmpi      conda-forge
    openmpi: 4.0.1-hc99cbb1_2 local     

when it should have libgfortran according to the strong run_exports on gfortran_linux-aarch64, since it has a build dependency on all the compilers (c, cxx, fortran).

[ax3l]

But we are not yet using conda-build 3.18.7 but instead 3.17.8 in the linked PR, no?

[minrk]

That PR's using 3.18.7

[xhochy]

Another failure is https://github.com/conda-forge/icu-feedstock/pull/25/checks?check_run_id=170794956 in conda-forge/icu-feedstock#25

[jjhelmus]

Closing this particular issue since the lack of channeldata.json has been fixed. The related issue where different builds/hashes express different run_exports can be discussed in #3617.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically locked because it has not had recent activity after being closed.

Please open a new issue if needed.

Thanks!