Build wheels with conda-build

[jjhelmus]

Add a --wheel argument to the 'conda build' command that when included uses
conda build to create and test a Python wheel file.

closes #1294

[jjhelmus]

The following meta.yaml file (based on the conda-forge jinja2-feedstock) can be used to create a Python wheel for Jinja2:

{% set version="2.8" %}

package:
  name: jinja2
  version: {{ version }}

source:
  fn: Jinja2-{{ version }}.tar.gz
  url: https://pypi.io/packages/source/J/Jinja2/Jinja2-{{ version }}.tar.gz
  md5: edb51693fe22c53cee5403775c71a99e

build:
  script: python setup.py bdist_wheel

requirements:
  build:
    - python
    - setuptools
    - wheel
    - markupsafe
  run:
    - python
    - setuptools
    - wheel
    - markupsafe

test:
  imports:
    - jinja2

about:
  home: http://jinja.pocoo.org
  license: 3-Clause BSD

The build is run as follows:

$ conda build --wheel .
BUILD START: jinja2-2.8-py35_0
    (actual version deferred until further download or env creation)
...
===== testing package: jinja2-2.8-py35_0 =====
import: 'jinja2'
===== jinja2-2.8-py35_0 OK =====
TEST END: jinja2-2.8-py35_0
INFO:/Users/jhelmus/dev/conda-forge/conda-build/conda_build/config.py:--keep-old-work flag not specified.  Removing source and build files.

# If you want to upload this package to anaconda.org later, type:
#
# $ anaconda upload /Users/jhelmus/anaconda/conda-bld/osx-64/Jinja2-2.8-py2.py3-none-any.whl
#
# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes

[jjhelmus]

A preprocessing selector for "wheel" builds would be a good addition if this is accepted as is so wheel only requirements or build scripts can be specified in the recipe metadata.

[msarahan]

Very interesting. I think we have quite a bit of work before this is final, but it's a great start. In particular:

• I'd like it if the setup command did not have to change, somehow. Rather, that conda-build provided the bdist_wheel bit when building for a wheel
• I think it would be better to add the whl_build argument as a new part of the config, rather than an argument. You can pass keyword arguments to the Config constructor, or to get_or_merge_config, and they'll be stored as attributes. In particular, by adding a new argument to the api build function, I feel like I've promised to bump the major version.
• I'm itching to break up output types into modular options, but I might be able to restrain myself for a little while.

Thanks for putting this forward! Do you want to iterate on it here more, or would you prefer that I merge and take over?

[jjhelmus]

I'm happy to iterate on this PR. I agree that there are many things than can (and should) be corrected before this is finalized.

I'd like it if the setup command did not have to change, somehow. Rather, that conda-build provided the bdist_wheel bit when building for a wheel

I would like this too, but I think there are cases where just running python setup.py bdist_wheel is not sufficient and being able to run a build script with all the capabilities conda build provides will be a great help. Some examples I can think of are:

• Setting pre-build environment variables or flags.
• Running auditwheel repair or delocate to include additional compiled libraries in the wheel file.

Some of utilities could be included in conda build itself but I think a more flexible (and likely maintainable) options is to allow a build script to call out to other tools that might be needed.

Perhaps adding a wheel_script entry to the build section of the yaml would be logical. If this was missing then python setup.py bdist_wheel would be used. As I mentioned in an earlier comment a whl selector would be another option.

I think it would be better to add the whl_build argument as a new part of the config, rather than an argument. You can pass keyword arguments to the Config constructor, or to get_or_merge_config, and they'll be stored as attributes. In particular, by adding a new argument to the api build function, I feel like I've promised to bump the major version.

Agreed, I'll work on changing this. Do you have an issue or alternative to storing the path to the produced whl file in the config?

[msarahan]

Some of utilities could be included in conda build itself but I think a more flexible (and likely maintainable) options is to allow a build script to call out to other tools that might be needed.

Yes, definitely. The tricky part is when the external tool gets called. There can and probably should be several different hooks. For python, I think we can get away with jinja stuff pretty well:

in meta.yaml:

build:
  script:
    - python setup.py {{ SETUP_PY_COMMAND }}

and we can have conda-build set that to install, or (install + setuptools junk) or bdist wheel, as necessary.

Now, the interesting question is whether the collection of files is something that should be packaged up (this is our current assumption), or something that is a package itself, and should be passed through.

Do you have an issue or alternative to storing the path to the produced whl file in the config?

Not especially, though I think answering the question of whether the list of changed files is a collection of things to be packaged or actual packages to be collected sort of obviates this.

[jjhelmus]

I adjusted the wheel building logic to store the --wheel flag setting in the config object which preserves the currently functions in conda_build.api.

Added a wheel selector which can be used in the meta.yaml file to specify lines which should be included only for wheel builds (or non-wheel builds with the [not wheel] selector). This allows a single meta.yaml to be used to build both wheels and conda packages. For example:

{% set version="2.8" %}
{% set setup_py_command="bdist_wheel" %}  # [wheel]
{% set setup_py_command="install --single-version-externally-managed --record=record.txt" %}  # [not wheel]

package:
  name: jinja2
  version: {{ version }}

source:
  fn: Jinja2-{{ version }}.tar.gz
  url: https://pypi.io/packages/source/J/Jinja2/Jinja2-{{ version }}.tar.gz
  md5: edb51693fe22c53cee5403775c71a99e

build:
  script: python setup.py {{ setup_py_command }}

requirements:
  build:
    - python
    - setuptools
    - wheel        # [wheel]
    - markupsafe
  run:
    - python
    - setuptools
    - wheel        # [wheel]
    - markupsafe

test:
  imports:
    - jinja2

about:
  home: http://jinja.pocoo.org
  license: 3-Clause BSD

[jjhelmus]

Now, the interesting question is whether the collection of files is something that should be packaged up (this is our current assumption), or something that is a package itself, and should be passed through.

A number of output formats (wheels, rpms, deb) produce a single file archive which is a collection of files. I'm guessing conda build may in the future want to to produce and handle these formats so being able to pass these file through seems to make sense to me. Copying them somewhere under the conda build output directory also makes sense to me, although the same directory as as the conda packages, as this PR is currently doing, might not be the best solutions. Would a CONDA_BLD_PATH/whl directory be more reasonable?

[msarahan]

Superseded by #1576 - @jjhelmus please take a look there and let me know what you think. I'm writing up a docs PR now.

[msarahan]

Docs PR at conda/conda-docs#388

[github-actions]

Hi there, thank you for your contribution!

This pull request has been automatically locked because it has not had recent activity after being closed.

Please open a new issue or pull request if needed.

Thanks!