automated build tweaks

[daler]

No description provided.

[daler]

It was taking a while for me to download packages for the CentOS container (firewall issues at work I think?) so I added a Dockerfile to build a locally-cached container, an additional script to support it, and docs.

Also, I realized the existing update_binstar_packages.py script was trying to build osx-64 packages from within the CentOS container. As I understand it, cross-platform building only works for pure-Python packages. of which there are relatively few in this repo. So I changed the build script to only build for the platform the script is run on.

Last, I found it useful to force builds when debugging this stuff, so I added a --force-build option.

Rather than push the changes I figured I'd use a PR to double check with everyone else to see if this makes sense.

[chapmanb]

These two lines should go in update_binstar_packages_docker_prebuilt.sh -- they turn on gcc-4.8 which is helpful for building a lot of packages. I'm not sure how well this gets cached between a recipe and a run.

[chapmanb]

Ryan -- this is awesome, thank you.

• For the pre-built container, we could pre-build and host it on http://hub.docker.com to make this easier for folks to just pull in.
• For building OSX packages, it should only try to do this for a Python build (with setup.py in the shell script) and should fail automatically if there are C packages. It is pretty useful to have the cross-compile build and I could use this functionality if I migrate python-only things over from bcbio. We could also have an option to do this as well, and have it default to only building for the current platform.

Thanks again.

[daler]

Brad -- thanks for your comments.

Pre-building the docker container sounds good. @johanneskoester do you have plans for a bioconda dockerhub account?

For the OSX packages, it was working as you said. The automatic failure came at the end of building pysam, which took a while to build, prompting me to restrict to Linux. I'll put the cross-compile build functionality back in by default to facilitate future bcbio python-only packages.

It would be nice to detect whether or not a Python package has C/C++ packages so the build fails early. I'm thinking presence of .pyx, .c, .cxx, or .cpp files and the string Extension in setup.py. Can you think of cases where this would give false positives or false negatives?

(also: as you suggested in the commit comments, I moved the gcc-4.8-related lines from Dockerfile to update_binstar_packages_docker_prebuilt.sh.)

[johanneskoester]

Great work!
I wonder if we could set up kind of a continuous integration build process via the Docker image and Travis CI:
http://docs.travis-ci.com/user/docker/

Apart from that, a bioconda docker hub account is a good idea anyway, because we need to pull the image from there also in case of using travis.

What do you think?

[johanneskoester]

Hi guys,
I have created a dockerhub organization for bioconda and added you.

[daler]

Johannes, thanks for the dockerhub account.

And very cool, I didn't realize Docker can be used on travis-ci now. CI builds would be great to have.

From my reading of the travis-ci docs, I think it would be possible to use encrypted environmental variables to deploy successful builds to anaconda.

[johanneskoester]

Ok. Let me start with some relevant links:

creating binstar token and encrypt it for travis CI:
https://github.com/rmcgibbo/python-appveyor-conda-example#binstar

installing miniconda in travis:
http://conda.pydata.org/docs/travis.html

[chapmanb]

Ryan and Johannes;
Awesome work, thank you. I pushed a Docker image to the repository hub so we should be good to go with using this for builds. I tested on a new version of cutadapt. I also updated the README to use this as the default approach for Linux. Let me know if you run into any problems at all. Looking forward to seeing what you come up with using Travis CI magic. Thanks again.