Migration to conda-build 3 and new compilers from Anaconda

[msarahan]

Mega-thread for discussion and issue tracking with migration.

Currently known things to address:

• how does toolchain package translate to compiler activation flags with new compilers?
• Pinning is managed with conda_build_config.yaml files in conda-build 3. How to migrate conda-forge's existing pinning scheme to that?
• Compatibility of Anaconda compiler's flags with conda-forge's existing defaults: are the default flags in the anaconda packages good enough that conda-forge need not carry its own customization? Note: customization risks incompatibility.
• conda-build-all and job computation. CB3 has native matrix support, and outputs jobs as rendered MetaData objects. In Anaconda's CI system, we dump the conda_build_config.yaml files from those rendered objects, and use those dumped conda_build_config.yaml files to build the original templated recipe. We probably need a way to output temporary folders for the CI's to process as jobs. CB3's matrix support does not use env vars (but could).
• Decide on usage of run_exports. run_exports essentially is the feature where a package declares what its runtime needs are if anyone uses that package at build time. This feature can make pinning much simpler, but perhaps makes recipes harder to read, because runtime dependencies become indirectly specified. Anaconda's recipes use run_exports very liberally.

Added 10/18/2017:

• Implement windows activation script wrappers that are compatible with Appveyor's VS situation

Lots more I'm probably missing here, so please comment and/or modify this issue.

[ocefpaf]

1 and 3 are kind of related and I would love to hear @isuruf's thoughts on those.

2 is, IMO, a no brainier. We kill the current scheme, serve a canonical conda_build_config.yaml file somewhere (maybe the conda build setup will be useful for this) and, in some rare cases, use a local file to override the default one.

It kind of solves the bot issue we have at the moment but we would need to trigger rebuilds when conda_build_config.yaml is updated. Or just let things "break" and re-build as needed (not so different from what happens right now with the dead bot).

A long term solution would be a service that rebuilds stuff following the dependency tree. One of my ideas for the Google Summer of Code 2018 BTW.

I don't have an option on 4 besides outsourcing as much as we can from conda-build-all to conda-build.

[isuruf]

how does toolchain package translate to compiler activation flags with new compilers?

I think we can remove toolchain. Do the new compilers have flags like -I${PREFIX}/include and -Wl,-rpath,${PREFIX}/lib ?

[msarahan]

I don't think those are defaults. I have needed to add them to some recipes. @mingwandroid, can you tell us why they are not defaults? Are they unsafe in some cases?

[mingwandroid]

I would rather not add -I${PREFIX}/include or -L${PREFIX}/lib because that un-generalizes the compilers.

[mingwandroid]

Same for -Wl,-rpath,... these things should be added when needed.

[isuruf]

I would rather not add -I${PREFIX}/include or -L${PREFIX}/lib because that un-generalizes the compilers.

I always think of $PREFIX as similar to /usr where the system compiler gcc checks for headers and libraries. In that case it makes sense for the gcc compiler in a conda-build environment to check in $PREFIX. This doesn't have to be in a normal conda environment, but can be done in a conda-build environment, just like we disable setuptools downloading in a conda-build environment. (I don't know anything about the new compilers, so I don't have any idea how this can be done)

[mingwandroid]

Adding a check for $CONDA_BUILD would be 'ok' in my opinion, still I do not like passing flags to builds that do not need them and serve no purpose.

[msarahan]

I would like to add a check for $CONDA_BUILD and add these flags when true. When builds fail due to missing this, it is an annoying hurdle for people to learn about. I will try to add this into conda-build today.

[jakirkham]

By adding into conda-build, do you mean setting environment variables for header and library search paths? If so, strong +1 from me. 😄

[msarahan]

In the activate.sh scripts provided by the compiler wrapper packages, gcc_linux-64 & clang_osx-64

if [ "$CONDA_BUILD" == "1" ]; then
    CFLAGS="$CFLAGS -I$PREFIX/include"
    LDFLAGS="$LDFLAGS -L$PREFIX/lib -Wl,-rpath,$PREFIX/lib"
fi

CXXFLAGS would be set equivalently in activate scripts for the CXX packages.

(open to suggestions on reordering arguments)

[msarahan]

So, I guess to clarify, this isn't going to be set by conda-build, per se. Conda-build will set CONDA_BUILD, and then the compiler activate scripts will behave a little differently to add these flags.

[mingwandroid]

A good example of where -I${PREFIX}/include and -L${PREFIX}/lib does cause real trouble is when you have a dependency on a package that depends on libiconv on linux. This package should not exist on linux since glibc implements iconv and libiconv was extracted from it.

Adding those flags causes an undeclared, unnecessary and incorrect dependency to creep in to your build. If the package that did depend on libiconv is later corrected so that it doesn't depend on libiconv, you end up with a broken package that links to iconv symbols it never should've linked to (and worse, has a DT_NEEDED entry of libiconv.so).

So I'm still reluctantly saying 'ok', but it's really wrong IMHO.

[msarahan]

Yeah, too bad there's no completely clear win here. I think the benefits of adding it outweigh the costs, but there also needs to be a way to disable it.

[mingwandroid]

Maybe we could delay such a change until conda-build/pyldd is equipped to warn (or better, error) on undeclared dependencies creeping into DT_NEEDED?

FWIW, to me it feels like we add those include and link FLAGS to far fewer than half of the recipes in AD5 (but I haven't checked).

[msarahan]

Half of the recipes in AD5 is a lot. If this trips up even 1/100 of the recipes in conda-forge, it's a big waste of people's time. I'd much rather fix the few recipes that this does affect negatively (these seem to be at the bottom of the stack), and have stuff further up the stack "just work"

[isuruf]

There are 198 recipes in conda-forge adding -L${PREFIX}/lib which is like 5% of all the recipes or 25% of the recipes with compilation.

[ocefpaf]

A good example of where -I${PREFIX}/include and -L${PREFIX}/lib does cause real trouble is when you have a dependency on a package that depends on libiconv on linux. This package should not exist on linux since glibc implements iconv and libiconv was extracted from it.

@mingwandroid it does look like the libiconv case is a pathological one. (BTW I did not know that, we added libiconv everywhere to satisfy some old scientific software and maybe we should revisit its need.)

However, I am more on @isuruf's side. It does looks natural to conda do add its PREFIX by default.
With that said, I kind of got used to add that manually b/c that is what we've been doing since day one. So I am +0.5.

[isuruf]

CXXFLAGS would be set equivalently in activate scripts for the CXX packages.

At toolchain, we are setting CPPFLAGS also

[isuruf]

Pinning is managed with conda_build_config.yaml files in conda-build 3. How to migrate conda-forge's existing pinning scheme to that?

Can conda-build 3 do what run_exports is doing using a config.yaml file? If so, it'll be easier to migrate. From that I mean, we pin in the build section and not run where the runtime pinning is based on the buildtime pinning. That way we can update pins in recipes.

Other option would be to remove pinning in recipes and have config.yaml do that. Some issues I have with it are,

1. If there is a dependency that doesn't need to be pinned what happens? For eg: we pin boost-cpp, but if a package needs only the boost headers, but not the libraries, what happens?
2. When we update a pinning, right now we look at the individual recipes and update them which is done manually or through a bot. If we switched to a config.yaml, we'll have no idea which recipe needs to be rebuilt or not. (For Anaconda, this is less of an issue because from what I understand, you rebuild everything by looking at dependencies when a pinning change happens, but conda-forge is limited by CI)

[notestaff]

"-I${PREFIX}/include and -L${PREFIX}/lib does cause real trouble is when you have a dependency on a package that depends on libiconv on linux" -- could the lint checker catch such errors?

[msarahan]

For anyone who wants an example of run_exports and centralized pinning, I've been working on libgdal. here's the recipe: https://github.com/AnacondaRecipes/libgdal-feedstock/blob/master/recipe/meta.yaml

Note that there's only one run requirement. The built package gets everything else from its build dependencies' run_exports. The output package metadata looks like:

{
  "arch": "x86_64",
  "build": "h28942fd_0",
  "build_number": 0,
  "depends": [
    "curl",
    "expat >=2.2.4,<3.0a0",
    "freexl >=1.0.4,<2.0a0",
    "geos >=3.6.0,<3.6.1.0a0",
    "giflib >=5.1.4,<5.2.0a0",
    "hdf4 >=4.2.13,<4.2.14.0a0",
    "hdf5 >=1.10.1,<1.10.2.0a0",
    "jpeg >=9b,<10a",
    "kealib >=1.4.7,<1.5.0a0",
    "libdap4 >=3.19.0,<3.20.0a0",
    "libkml >=1.3.0,<1.4.0a0",
    "libnetcdf >=4.4.1.1,<4.4.2.0a0",
    "libpng >=1.6.32,<1.7.0a0",
    "libpq >=9.6.5",
    "libspatialite >=4.3.0a,<4.4.0a0",
    "libtiff >=4.0.8,<5.0a0",
    "openjpeg >=2.2.0,<3.0a0",
    "poppler >=0.60.1,<1.0a0",
    "proj4 >=4.9.3,<4.9.4.0a0",
    "sqlite >=3.20.1,<4.0a0",
    "xerces-c >=3.2.0,<3.3.0a0",
    "xz >=5.2.3,<6.0a0",
    "zlib >=1.2.11,<1.3.0a0"
  ],
  "license": "MIT",
  "name": "libgdal",
  "platform": "osx",
  "subdir": "osx-64",
  "timestamp": 1508362884739,
  "version": "2.2.2"
}

The actual versions of those inputs come from https://github.com/AnacondaRecipes/aggregate/blob/master/conda_build_config.yaml

Note that there are 2 hdf5 versions in that conda_build_config.yaml file - building this libgdal recipe results in 2 output packages - one built for each hdf5 version.

[isuruf]

@msarahan, thanks. That's great.
I have two questions.

1. Is it possible to ignore run_exports?. For example a python package which loads a dynamic library using cffi don't need any pinning.

2. Is it possible to get this output metadata without actually building the package? For the maintenance bot, what we could do is get the output metadata from the latest built package and compare the output metadata if the package was built right now and if not trigger a rebuild.

[msarahan]

1. Yes (from https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#pin-downstream):

The potential downside of this feature is that it takes some control over constraints away from downstream users. If an upstream package has a problematic run_exports constraint, you can ignore it in your recipe by listing the upstream package name in the build/ignore_run_exports section:

build:
  ignore_run_exports:
    - libstdc++

2. Yes, conda render takes all run_exports information into account. Here's the outputs from libgdal (note 2 recipes, due to the 2 hdf5 variants):

$ conda render libgdal-feedstock
package:
    name: libgdal
    version: 2.2.2
source:
    patches:
        - disable_jpeg12.patch
        - windowshdf5.patch
        - multiprocessor.patch
    sha256: 14c1f78a60f429ad51c08d75cbf49771f1e6b20e7385c6e8379b40e8dfa39544
    url: http://download.osgeo.org/gdal/2.2.2/gdal-2.2.2.tar.gz
build:
    number: '0'
    run_exports:
        - libgdal >=2.2.2,<2.3.0a0
requirements:
    build:
        - glib 2.53.6 ha08cb78_1
        - ld64 274.2 h7c2db76_0
        - sqlite 3.20.1 h900c3b0_1
        - openssl 1.0.2l h6cc03a0_4
        - expat 2.2.4 hf840079_2
        - geos 3.6.0 hd02dd01_1
        - gettext 0.19.8.1 hb0f4f8b_2
        - icu 58.2 hea21ae5_0
        - jpeg 9b haccd157_1
        - hdf5 1.8.18 h1edb405_0
        - openjpeg 2.2.0 hede6aa5_1
        - zlib 1.2.11 h60db283_1
        - postgresql 9.6.5 hf30a1d1_0
        - libspatialite 4.3.0a h3ada6ff_17
        - pcre 8.41 h29eefc5_0
        - freexl 1.0.4 hc15b6b0_3
        - giflib 5.1.4 h6b45fa3_1
        - clang 4.0.1 h662ec87_0
        - xz 5.2.3 h808ee3b_1
        - pixman 0.34.0 h33cc98f_2
        - ca-certificates 2017.08.26 ha1e5d58_0
        - bzip2 1.0.6 h92991f9_1
        - proj4 4.9.3 h8d5df23_6
        - hdf4 4.2.13 h0cfac3f_1
        - curl 7.55.1 h0601bc4_3
        - freetype 2.8 h143eb01_0
        - libdap4 3.19.0 h57690e0_1
        - cctools 895 h7512d6f_0
        - libcxx 4.0.1 h579ed51_0
        - libpng 1.6.32 hce72d48_2
        - poppler 0.60.1 h25db2b2_0
        - libffi 3.2.1 hd939716_3
        - libcxxabi 4.0.1 hebd6815_0
        - cairo 1.14.10 h842a265_5
        - clang_osx-64 4.0.1 hc6fb1d2_0
        - compiler-rt 4.0.1 h5487866_0
        - libxml2 2.9.4 hbd0960b_5
        - libedit 3.1 hb4e282d_0
        - libnetcdf 4.4.1.1 h185d453_7
        - ncurses 6.0 ha932d30_1
        - xerces-c 3.2.0 h4094662_0
        - json-c 0.12.1 h7fa5a77_1
        - llvm 4.0.1 hc748206_0
        - kealib 1.4.7 h20aded0_4
        - libboost 1.65.1 h0af5023_1
        - libkml 1.3.0 he0c3ce7_2
        - libtiff 4.0.8 h8cd0352_9
        - poppler-data 0.4.8 hf2eda46_0
        - libgfortran 3.0.1 h93005f0_2
        - libssh2 1.8.0 h1218725_2
        - fontconfig 2.12.4 h30cff1d_2
        - libiconv 1.15 he69fb81_6
        - llvm-lto-tapi 4.0.1 h6701bc3_0
    host:
        - glib 2.53.6 ha08cb78_1
        - sqlite 3.20.1 h900c3b0_1
        - openssl 1.0.2l h6cc03a0_4
        - expat 2.2.4 hf840079_2
        - geos 3.6.0 hd02dd01_1
        - gettext 0.19.8.1 hb0f4f8b_2
        - icu 58.2 hea21ae5_0
        - jpeg 9b haccd157_1
        - hdf5 1.8.18 h1edb405_0
        - openjpeg 2.2.0 hede6aa5_1
        - zlib 1.2.11 h60db283_1
        - postgresql 9.6.5 hf30a1d1_0
        - libspatialite 4.3.0a h3ada6ff_17
        - pcre 8.41 h29eefc5_0
        - freexl 1.0.4 hc15b6b0_3
        - giflib 5.1.4 h6b45fa3_1
        - xz 5.2.3 h808ee3b_1
        - pixman 0.34.0 h33cc98f_2
        - ca-certificates 2017.08.26 ha1e5d58_0
        - bzip2 1.0.6 h92991f9_1
        - proj4 4.9.3 h8d5df23_6
        - hdf4 4.2.13 h0cfac3f_1
        - curl 7.55.1 h0601bc4_3
        - freetype 2.8 h143eb01_0
        - libdap4 3.19.0 h57690e0_1
        - libcxx 4.0.1 h579ed51_0
        - libpng 1.6.32 hce72d48_2
        - poppler 0.60.1 h25db2b2_0
        - libffi 3.2.1 hd939716_3
        - libcxxabi 4.0.1 hebd6815_0
        - cairo 1.14.10 h842a265_5
        - libxml2 2.9.4 hbd0960b_5
        - libedit 3.1 hb4e282d_0
        - libnetcdf 4.4.1.1 h185d453_7
        - ncurses 6.0 ha932d30_1
        - xerces-c 3.2.0 h4094662_0
        - json-c 0.12.1 h7fa5a77_1
        - kealib 1.4.7 h20aded0_4
        - libboost 1.65.1 h0af5023_1
        - libkml 1.3.0 he0c3ce7_2
        - libtiff 4.0.8 h8cd0352_9
        - poppler-data 0.4.8 hf2eda46_0
        - libgfortran 3.0.1 h93005f0_2
        - libssh2 1.8.0 h1218725_2
        - fontconfig 2.12.4 h30cff1d_2
        - libiconv 1.15 he69fb81_6
    run:
        - giflib >=5.1.4,<5.2.0a0
        - libtiff >=4.0.8,<5.0a0
        - kealib >=1.4.7,<1.5.0a0
        - jpeg >=9b,<10a
        - sqlite >=3.20.1,<4.0a0
        - hdf4 >=4.2.13,<4.2.14.0a0
        - libpng >=1.6.32,<1.7.0a0
        - libdap4 >=3.19.0,<3.20.0a0
        - proj4 >=4.9.3,<4.9.4.0a0
        - poppler >=0.60.1,<1.0a0
        - expat >=2.2.4,<3.0a0
        - curl
        - libpq  >=9.6.5
        - zlib >=1.2.11,<1.3.0a0
        - hdf5 >=1.8.18,<1.8.19.0a0
        - libspatialite >=4.3.0a,<4.4.0a0
        - geos >=3.6.0,<3.6.1.0a0
        - openjpeg >=2.2.0,<3.0a0
        - libnetcdf >=4.4.1.1,<4.4.2.0a0
        - freexl >=1.0.4,<2.0a0
        - xerces-c >=3.2.0,<3.3.0a0
        - libkml >=1.3.0,<1.4.0a0
        - xz >=5.2.3,<6.0a0
test:
    commands:
        - gdal_grid --version
        - gdal_rasterize --version
        - gdal_translate --version
        - gdaladdo --version
        - gdalenhance --version
        - gdalwarp --version
        - gdalinfo --formats
        - gdalinfo http://thredds.nersc.no/thredds/dodsC/greenpath/Model/topaz
        - conda inspect linkages -p $PREFIX libgdal
        - conda inspect objects -p $PREFIX libgdal
    files:
        - test_data
about:
    home: http://www.gdal.org/
    license: MIT
    license_file: LICENSE.TXT
    summary: The Geospatial Data Abstraction Library (GDAL).
extra:
    copy_test_source_files: true
    final: true
    recipe-maintainers:
        - ocefpaf
        - kmuehlbauer
        - gillins
        - msarahan

package:
    name: libgdal
    version: 2.2.2
source:
    patches:
        - disable_jpeg12.patch
        - windowshdf5.patch
        - multiprocessor.patch
    sha256: 14c1f78a60f429ad51c08d75cbf49771f1e6b20e7385c6e8379b40e8dfa39544
    url: http://download.osgeo.org/gdal/2.2.2/gdal-2.2.2.tar.gz
build:
    number: '0'
    run_exports:
        - libgdal >=2.2.2,<2.3.0a0
requirements:
    build:
        - glib 2.53.6 ha08cb78_1
        - ld64 274.2 h7c2db76_0
        - sqlite 3.20.1 h900c3b0_1
        - openssl 1.0.2l h6cc03a0_4
        - expat 2.2.4 hf840079_2
        - geos 3.6.0 hd02dd01_1
        - gettext 0.19.8.1 hb0f4f8b_2
        - icu 58.2 hea21ae5_0
        - jpeg 9b haccd157_1
        - openjpeg 2.2.0 hede6aa5_1
        - libspatialite 4.3.0a h3ada6ff_17
        - zlib 1.2.11 h60db283_1
        - postgresql 9.6.5 hf30a1d1_0
        - pcre 8.41 h29eefc5_0
        - freexl 1.0.4 hc15b6b0_3
        - giflib 5.1.4 h6b45fa3_1
        - clang 4.0.1 h662ec87_0
        - xz 5.2.3 h808ee3b_1
        - pixman 0.34.0 h33cc98f_2
        - ca-certificates 2017.08.26 ha1e5d58_0
        - bzip2 1.0.6 h92991f9_1
        - proj4 4.9.3 h8d5df23_6
        - hdf4 4.2.13 h0cfac3f_1
        - curl 7.55.1 h0601bc4_3
        - freetype 2.8 h143eb01_0
        - libnetcdf 4.4.1.1 h651529c_7
        - libdap4 3.19.0 h57690e0_1
        - cctools 895 h7512d6f_0
        - libcxx 4.0.1 h579ed51_0
        - libpng 1.6.32 hce72d48_2
        - poppler 0.60.1 h25db2b2_0
        - libffi 3.2.1 hd939716_3
        - libcxxabi 4.0.1 hebd6815_0
        - cairo 1.14.10 h842a265_5
        - clang_osx-64 4.0.1 hc6fb1d2_0
        - compiler-rt 4.0.1 h5487866_0
        - libxml2 2.9.4 hbd0960b_5
        - libedit 3.1 hb4e282d_0
        - kealib 1.4.7 h73e85dd_4
        - ncurses 6.0 ha932d30_1
        - xerces-c 3.2.0 h4094662_0
        - json-c 0.12.1 h7fa5a77_1
        - llvm 4.0.1 hc748206_0
        - hdf5 1.10.1 h6090a45_0
        - libboost 1.65.1 h0af5023_1
        - libkml 1.3.0 he0c3ce7_2
        - libtiff 4.0.8 h8cd0352_9
        - poppler-data 0.4.8 hf2eda46_0
        - libgfortran 3.0.1 h93005f0_2
        - libssh2 1.8.0 h1218725_2
        - fontconfig 2.12.4 h30cff1d_2
        - libiconv 1.15 he69fb81_6
        - llvm-lto-tapi 4.0.1 h6701bc3_0
    host:
        - glib 2.53.6 ha08cb78_1
        - sqlite 3.20.1 h900c3b0_1
        - openssl 1.0.2l h6cc03a0_4
        - expat 2.2.4 hf840079_2
        - geos 3.6.0 hd02dd01_1
        - gettext 0.19.8.1 hb0f4f8b_2
        - icu 58.2 hea21ae5_0
        - jpeg 9b haccd157_1
        - openjpeg 2.2.0 hede6aa5_1
        - libspatialite 4.3.0a h3ada6ff_17
        - zlib 1.2.11 h60db283_1
        - postgresql 9.6.5 hf30a1d1_0
        - pcre 8.41 h29eefc5_0
        - freexl 1.0.4 hc15b6b0_3
        - giflib 5.1.4 h6b45fa3_1
        - xz 5.2.3 h808ee3b_1
        - pixman 0.34.0 h33cc98f_2
        - ca-certificates 2017.08.26 ha1e5d58_0
        - bzip2 1.0.6 h92991f9_1
        - proj4 4.9.3 h8d5df23_6
        - hdf4 4.2.13 h0cfac3f_1
        - curl 7.55.1 h0601bc4_3
        - freetype 2.8 h143eb01_0
        - libnetcdf 4.4.1.1 h651529c_7
        - libdap4 3.19.0 h57690e0_1
        - libcxx 4.0.1 h579ed51_0
        - libpng 1.6.32 hce72d48_2
        - poppler 0.60.1 h25db2b2_0
        - libffi 3.2.1 hd939716_3
        - libcxxabi 4.0.1 hebd6815_0
        - cairo 1.14.10 h842a265_5
        - libxml2 2.9.4 hbd0960b_5
        - libedit 3.1 hb4e282d_0
        - kealib 1.4.7 h73e85dd_4
        - ncurses 6.0 ha932d30_1
        - xerces-c 3.2.0 h4094662_0
        - json-c 0.12.1 h7fa5a77_1
        - hdf5 1.10.1 h6090a45_0
        - libboost 1.65.1 h0af5023_1
        - libkml 1.3.0 he0c3ce7_2
        - libtiff 4.0.8 h8cd0352_9
        - poppler-data 0.4.8 hf2eda46_0
        - libgfortran 3.0.1 h93005f0_2
        - libssh2 1.8.0 h1218725_2
        - fontconfig 2.12.4 h30cff1d_2
        - libiconv 1.15 he69fb81_6
    run:
        - giflib >=5.1.4,<5.2.0a0
        - libtiff >=4.0.8,<5.0a0
        - kealib >=1.4.7,<1.5.0a0
        - jpeg >=9b,<10a
        - sqlite >=3.20.1,<4.0a0
        - hdf4 >=4.2.13,<4.2.14.0a0
        - libpng >=1.6.32,<1.7.0a0
        - libdap4 >=3.19.0,<3.20.0a0
        - proj4 >=4.9.3,<4.9.4.0a0
        - poppler >=0.60.1,<1.0a0
        - expat >=2.2.4,<3.0a0
        - hdf5 >=1.10.1,<1.10.2.0a0
        - curl
        - libpq  >=9.6.5
        - zlib >=1.2.11,<1.3.0a0
        - libspatialite >=4.3.0a,<4.4.0a0
        - geos >=3.6.0,<3.6.1.0a0
        - openjpeg >=2.2.0,<3.0a0
        - libnetcdf >=4.4.1.1,<4.4.2.0a0
        - freexl >=1.0.4,<2.0a0
        - xerces-c >=3.2.0,<3.3.0a0
        - libkml >=1.3.0,<1.4.0a0
        - xz >=5.2.3,<6.0a0
test:
    commands:
        - gdal_grid --version
        - gdal_rasterize --version
        - gdal_translate --version
        - gdaladdo --version
        - gdalenhance --version
        - gdalwarp --version
        - gdalinfo --formats
        - gdalinfo http://thredds.nersc.no/thredds/dodsC/greenpath/Model/topaz
        - conda inspect linkages -p $PREFIX libgdal
        - conda inspect objects -p $PREFIX libgdal
    files:
        - test_data
about:
    home: http://www.gdal.org/
    license: MIT
    license_file: LICENSE.TXT
    summary: The Geospatial Data Abstraction Library (GDAL).
extra:
    copy_test_source_files: true
    final: true
    recipe-maintainers:
        - ocefpaf
        - kmuehlbauer
        - gillins
        - msarahan

[msarahan]

As for build triggering based on stuff already being built, that's actually a weak spot that's on my short-term list. The current implementation is an exact build string match. That's way too sensitive and ends up rebuilding when it's not necessary. It should factor in at least 3 things:

• name
• version
• build variables used

[jakirkham]

Like to add on the compiler front that it would be good to sort out how we handle/what we do with the existing LLVM based stack and anything that already relies on them. IOW llvmdev, clangdev, libcxx, openmp, and the pending flangdev. Also there is some discussion on adding LLVM's libunwind and have the non-GNU libunwind already packaged, which may need to be examined in this discussion.

cc @conda-forge/llvmdev @conda-forge/clangdev @conda-forge/libcxx @conda-forge/openmp

[mingwandroid]

Why is dev being added to all these package names? I cannot see any reason to do that. You wouldn't name gcc as gccdev for example. What's wrong with flang? Also why was openmp not identified with an llvm- prefix? Names are important and I'd like to know how I can throw my $0.02 in before these things get set in stone.

I have ended up having to create metapackages to point back to these names, e.g.:

https://github.com/AnacondaRecipes/boost-cpp-feedstock/blob/master/recipe/meta.yaml#L139-L159

With openmp, we might have wanted to use that name as a variant selecting metapackage but can't now.

[msarahan]

As Ray says, I recommend creating metapackages for these. They would point to compiler activation packages (e.g. clang_osx-64), which in turn point to "meat" compiler packages - the actual executables and all that.

The more we can agree on flags and centralize compiler packages (not duplicate packages on both conda-forge and defaults), the better (for the sake of cross-channel compatibility). I understand the arguments for keeping conda-forge self-contained, but I do feel like some compromise may be helpful here. We should discuss this in a call, though. Comments aren't really adequate.

What Ray is talking about regarding openmp is that there is more than one implementation. Intel also has one. Over time, it will be wise to ditch the existing meaning of the openmp package name, and use that name instead as a metapackage that can select different implementations.

[SylvainCorlay]

Names are important and I'd like to know how I can throw my $0.02 in before these things get set in stone.

The Debian community has very strict naming conventions for packages. For example, python packages are generally named python-pypi_package_name which prevents conflicts with ruby, perl and whatnot.

As conda becomes used more and more as a general-purpose package manager, instead of being focused on python, a conda-based distribution would benefit a lot from using such naming conventions.

[ghost]

We don't need to debate this here. It can wait for the meeting.

Unfortunately, I won't be able to attend the meeting. But what I will say is that this is not a debate that is happening here. Numpy will drop support and then everyone else will have to deal with it (that is unless people who don't bear the maintenance cost object to the plans).

Why is gfortran with msys2 not an option?

Scipy and Numpy have to ship CRT binaries to comply with the Python specification. This is not an issue that is up for debate, except on the python-dev mailing list. These projects will drop MSVC 2008 as the need to upgrade to C++11 becomes increasingly important. And that means dropping Python 2.7.

If you don't like this, then it's too bad. Because this is the future that python-dev has designed, and the decision goes all the way up to the BDFL. It's probably too late to undo at this point.

[ChrisBarker-NOAA]

OT, but briefly:

On Thu, Nov 9, 2017 at 1:43 PM, xoviat ***@***.***> wrote: Scipy and Numpy have to ship CRT binaries to comply with the Python specification. This is not an issue that is up for debate, except on the python-dev mailing list. sure it is -- the python.org build of 2.7 for Windows is built with MSVC

2008 -- and that is not going to change. But IF (and this is a big IF) there is a community demand for py2.7 with a newer compiler (either before or after 2.7 officially reaches end of life), then there is no technical reason Anaconda.com or conda-forge, (or whoever) couldn't provide a python2.7 built with a new and better compiler. That hasn't been done so far, as compatibility with the python.org builds has been considered important. But at some point, being able to build extensions, etc with a modern compiler may be compelling. However: there are a LOT of reasons to drop 2.7 support anyway -- getting a newer compiler on Windows should be the least of the reasons. So it looks like numpy is going to declare a drop date -- and we can all have one less "platform" to support, so I'll be happy for that day to come (even though I haven't ported my major projects yet!)

…

-CHB

[isuruf]

Anaconda does ship MinGW binaries. @mingwandroid is a major msys2 contributor. They might be a bit out of date now, but we have much of msys2 as conda packages.

Yes, but Anaconda keeps them separate, and use them rarely (like rpy2) when building a python extension.

Why is gfortran with msys2 not an option? Why is gfortran on Mac not an option?

gfortran on Mac is the best option and flang should be considered only for windows

[ghost]

@ChrisBarker-NOAA

I'm sure you already know, but I qualified my statement with "except on the python-dev mailing list." So essentially, what would have to happen is someone would have to start a thread to release a Python 2.8 with a newer ABI. Because projects are not going to use a Python that is not from Python.org.

Yes, theoretically it is up for debate (as I qualified). But not here. It's up for debate on python-dev. And it's up for debate in a very hostile environment that's (justifiably) decided on rules (PEP 404) that prohibit exactly the kinds of changes that would need to happen. I'll tip my hat to anyone who manages to get through that.

[ghost]

Why is gfortran on Mac not an option?

Also, I should point out that gfortran on mac is an option precisely because it is compatible with the native (clang) ABI. If gfortran on windows was compatible with the native (UCRT) ABI, then it would be an option on windows as well. But it's not, so it isn't.

[ChrisBarker-NOAA]

@xoviat:

Actually, that was my point -- python-dev has nothing to do with it -- if conda-forge or anaconda.org want to release a 2.7 built with a different compiler on Windows, they can do so -- it just won't be compatible with the python,org build.

This isn't all that different than the LInux situation -- plenty of binaries of pythonx.y that aren't binary compatible with each-other (at least before many linux...)

The community came to a consensus years ago that standardizing eh Windows build was the best way to go -- but times have changed -- MAYBE it's time for a new approach (NOTE: I"m not advocating for it -- I think there are other reasons to go to py3, so non need to stretch out support for py2)

as for "release a Python 2.8 with a newer ABI." - That was proposed a couple years ago by Christian Tismer for Stackless. It was soundly rejected.

Part of the point was that a 2.7 built with a different ABI is still 2.7, just like 2.7 on Mac, or Linux, or . is still 2.7. A new ABI is essentially a new platform, not a new python version.

And indeed, there is a (at least source) for a VS2010 build of 2.7:

https://github.com/ctismer/Python-2.7.5-VS2010

And as for: "projects are not going to use a Python that is not from Python.org. -- not so sure about that, plenty of people are building stuff on top of Anaconda/conda -- it's compatible with python.org, but it's not "from" python.org.

This could all be done -- but it would require a significant effort -- and I think that effort should be better spent on porting ot py3 -- then you get py3 and newer compilers for free!

(I still have no idea why Stackless hasn't gone to py3 -- wait! maybe it has:

https://github.com/stackless-dev/stackless

Which might be why Cristian gave up on "2.8"

[ghost]

Actually, that was my point -- python-dev has nothing to do with it -- if conda-forge or anaconda.org want to release a 2.7 built with a different compiler on Windows, they can do so -- it just won't be compatible with the python,org build.

Yes, but numpy and scipy will just ignore this build and drop support anyway. That's what I'm saying.

[ghost]

The moment that projects need to use C++11, they will drop support. The moment they need to use fortran, they will drop support. And then the cleanups begin. Removing all of the Python 2.7 hacks because "we don't need them anymore." And reworking the C API to take advantage of newer Python improvements. And then no matter what you do, you cannot build these projects on Python 2.7.

[mingwandroid]

While I like where flang is going, realistically it is probably 2 years away from being usable by any serious software distribution as their fortran compiler.

While I'm here I would like to clear up some misconceptions that have been thrown about regarding Microsoft C runtime compatibility.

Yes, mingw-w64 links to msvcrt.dll. You can load a DLL (such as a python module) that links to that into a Python that's linked to the vs2008 CRT or to the UCRT just fine. The only problem would be if you allocated something from a DLL linked to one C runtime and deallocated it in another or passed OS-level primitives between them. In reality this basically never happens as it would be very poor software design. I have certainly not see it.

This is not some theoretical musings, this is reasonably large part of my job. I have built a large amount of python modules that link to mingw-w64 compiled DLLs. Most recently libopus and libvpx linked to OpenCV, running on Python 2 and 3.

Also Theano dynamically compiles kernels as python modules using mingw-w64 and loads them dynamically.

It is entirely possible (and very easy) to use mingw-w64 with Cython.

Our RStudio does something very similar with a program called rsession.exe. It is compiled with VS2015 linking to the UCRT but the vast majority of what it does is done via the r-base dlls and those are all compiled with mingw-w64.

Relatedly, all of conda-forge's (and default's) R infrastructure on Windows is compiled with mingw-w64.

Another fine example of problem-free mingw-w64 integration with Python on Windows is XGBoost, the recipe for which can be seen at https://github.com/conda/conda-recipes/blob/master/xgboost

So I hope @xoviat has not managed to fool anyone into thinking this linking is impossible (or even highly problematic). Blatantly it is not.

To get on topic, most fortran software expects to be compiled with either gfortran or something that emulates gfortran's CLI. It is usually driven through Makefiles or autotools. Once flang is mainlined into llvm/clang and once it's been battle hardened by a few years of 'proper' use and once it emulates gfortran's CLI on Windows and once Python 2.7 has actually died, I will happily consider recommending it instead of gfortran.

Until then you people promoting flang over gfortran on Windows are just dreaming.

[ghost]

I agree with that; I even noted:

The only advantage for flang is that it's open source, and thus has a compatible license. Otherwise, it basically for now does almost everything that Intel does but performs worse.

My central claim was that projects will drop support for Python 2.7.

So I hope @xoviat has not managed to fool anyone into thinking this linking is impossible (or even highly problematic). Blatantly it is not.

Not only do I know this, and not only have I enabled SciPy to be built on Python 2.7 with this knowledge, but I have been quite devious. I have convinced a number of projects not to release Python 2.7 wheels on pypi.org for windows in order to inflict maximum pain on Python 2.7 windows users. Coincidentally, this has saved me a not-insignificant amount of effort in packaging these projects.

Until then you flang pushers are just dreaming.

I'll refer to my earlier statement. I don't disagree that flang is not the ideal compiler. But it gets the job done for many projects, which are usually written in F77 or F90 (flang has some problems with F2008).

The future that I see is that everything is basically compiled with MSVC/flang.

[mingwandroid]

So I hope @xoviat has not managed to fool anyone into thinking this linking is impossible (or even highly problematic). Blatantly it is not

Not only do I know this, and not only have I enabled SciPy to be build on Python 2.7 with this knowledge

Please show evidence of this claim. I backed up the claims I made with copious amounts of this, so far you have provided none.

[ghost]

For reference, the vtk package only has Python 3.6. scikit-umfpack only has Python 3.5 and Python 3.6. And there are even more projects similar to this.

[ghost]

I mean come on: scipy/scipy#7616

[mingwandroid]

For reference, the vtk package only has Python 3.6. scikit-umfpack only has Python 3.5 and Python 3.6. And there are even more projects similar to this.

.. because they use C++11 right? Or is it because you've convinced them not to bother? If it's either of those then that's got nothing to do with the issue of fortran though and I would ask that you please stop conflating them, credit your audience with some intelligence.

That you are taking and admitting to these acts of sabotage against Python 2.7 on Windows should rule you out from further involvement in this discussion as far as I am concerned. You are far from objective.

FWIW, I do not enjoy Python 2.7 on Windows, but I am not going to let that blind me to the correct course of action.

[ghost]

Or is it because you've convinced them not to bother?

No. It's because I chose not to package Python 2.7.

If it's either of those then that's got nothing to do with the issue of fortran though and I would ask that you please stop conflating them, credit your audience with some intelligence.

No, you're right that it has nothing to do with fortran. Again, I reiterate:

My central claim was that projects will drop support for Python 2.7.

That you are taking and admitting to these acts of sabotage against Python 2.7 on Windows should rule you out from further involvement in this discussion as far as I am concerned. You are far from objective.

You're correct. I am not objective. But I only have the same view as most of the core developers (@ChrisBarker-NOAA has the same view but just phrases it more nicely). And you might consider why I have this view: every single time I write code, it's always Python 2.7 that fails on the CI. Always. I'm sure that I'm not alone. And I'm biased precisely because I've had to spend extra time supporting a language that I've never used. My only goal is to make the development process seamless and Python 2.7 gets in the way of that.

[ghost]

Just look at what @wesm said. He agrees with my central point, except perhaps not the way I phrased it (@wesm you can't support c++14 and Python 2.7 without major hacks).

[mingwandroid]

Mmm, @wesm was talking about Linux, again, this is further conflation that doesn't help your arguments. To me it seems you are grasping at whatever you can (for it is entirely possible to use C++11, 14 and 1z on macOS and Linux in Python 2 modules using our toolchains).

Clearly your goals are not those of conda-forge who aims to build provide builds of software for Python 2 and Python 3 across Windows, macOS and Linux. I would ask you to refrain then from trying to sway this discussion.

[ghost]

And again, none of this discussion is going to change anything. What I think ultimately doesn't matter. It's what library maintainers and python developers think.

[mingwandroid]

Sure, if you convince various upstreams to stop supporting Python 2 or Python 2 on Windows then it will become too much work for us to undo your sabotage, but go to town if that's what you think you must do.

[mingwandroid]

This discussion is about conda-forge and conda-defaults unifying on the same major version of conda-build and the same compilers.

I do long for the day when flang can be used and python 2 is dead, but it's not today, and it's not next year either.

[mingwandroid]

long for the day

.. is a bit much; better to say I look forward to it!

[ghost]

Mmm, @wesm was talking about Linux, again, this is further conflation that doesn't help your arguments. To me it seems you are grasping at whatever you can.

Sorry, I meant Python 2.7 on windows. My mistake.

Clearly your goals are not those of conda-forge who aims to build provide builds of software for Python 2 and Python 3 across Windows, macOS and Linux. I would ask you to refrain then from trying to sway this discussion.

Again, what I think doesn't matter. I only care about this to the extent that I have a reasonable way to install a CRT compatible fortran compiler on windows. I'm going to head out at this point.

Sure, if you convince various upstreams to stop supporting Python 2 or Python 2 on Windows then it will become too much work for us to undo your sabotage, but go to town if that's what you think you must do.

I mean if projects drop support for Python 2 on windows, then Python 2 is dead on windows. Right?

.. is a bit much; better to say I look forward to it!

Then we're in agreement on this issue. It's just that I've taken concrete steps to achieve this goal.

[SylvainCorlay]

Just look at what @wesm said. He agrees with my central point, except perhaps not the way I phrased it (@wesm you can't support c++14 and Python 2.7 without major hacks).

@xoviat I have been building C++14 extensions for Python 2.7 for a while without issue.

On conda, adding a runtime dependency to the msvc14 runtime only on windows

vs2015_runtime # [win]

then populating the bld.bat with the call setting up the use of VS14 does the job.

if "%ARCH%" == "32" (set PLATFORM=x86) else (set PLATFORM=x64)
call "%VS140COMNTOOLS%\..\..\VC\vcvarsall.bat" %PLATFORM%
set DISTUTILS_USE_SDK=1
set MSSdk=1
"%PYTHON%" setup.py install
if errorlevel 1 exit 1

As said earlier by @mingwandroid, as soon as you don't deallocate memory allocated in one dll in another, (which is bad design), you should be fine with this.

(that said, you might be considering this a major hack.)

[msarahan]

@xoviat it seems like you did really nice work on the scipy wheels. Why did you use mingw there, but say it's not an option here? I agree that long-term flang will be the best option, but like Ray, I think it'll take time.

[ghost]

I agree that long-term flang will be the best option, but like Ray, I think it'll take time.

+1. That's always been my claim.

Why did you use mingw there, but say it's not an option here?

MinGW doesn't currently support reverse-symbol dependencies (although in theory it could). So you can have a fortran symbol that you need in C, but you can't have a C symbol that you need in fortran. Luckily, scipy doesn't have any C symbols that it needs in fortran, but that's not a seamless experience, which is the goal.

[ghost]

I will say this: if Anaconda can convince the Python core developers to release a theoretical Python 2.8, then I promise not to try to get Python 2 support dropped. @SylvainCorlay is correct. I knew about that option as well (although I admit that I did not consider it until he brought it up), but that's not a seamless experience either, because you can't just trust developers to do the right thing.

My goal is for people to run pip install . in a project source directory and have the project build. Not 50% of the time. Not 80% of the time, but nearly all of the time, and for dependencies to be installed with a package manager (conda happens to be the best option on windows) just like they are on OSX with homebrew and ubuntu with apt.

[ocefpaf]

Guys, this thread became unmanageable and unproductive. There are many good points here though that we cannot miss. B/c of that I am locking this thread and we should break it down later, after the meeting, into actionable topics.

PS: I for one would love to see a hacked 2.8 out there support modern compiler on Windows, but I don't think we, conda-forge, should provide that as the maintenance cost is beyond what we can bare.

[msarahan]

Compiler meeting results: Thursday, Nov 16 9AM Central looks to be the best time.
https://anaconda.webex.com/anaconda-en/j.php?MTID=m11b5ddad66325da22bbe58d7d1c02809

Agenda / notes at https://paper.dropbox.com/doc/Compiler-meeting-20171116-1lYaDIP93JRzzQZBlicxg?_tk=share_copylink