conda build scripts (fixes #608) #751

kain88-de · 2016-03-01T21:59:11Z

Fixes #608

Changes made in this Pull Request:

Added conda scripts

This updates the scripts from #571 by @hainm. I currently uploaded
them on my own account. I still have to figure out how to do this for
the MDAnalysis org once I have access to it.

https://anaconda.org/kain88-de/mdanalysis

PR Checklist

CHANGELOG updated?
Issue raised/referenced?
- [ ] publish blog post detailing how MDAnalysis can be installed from conda
add anaconda badge
- [ ] add instructions to Quick Start Installation

EDIT

I remove some bullet points because they are tracked here

kain88-de · 2016-03-03T22:43:35Z

So I'm having a problem writing the build scripts that also run the tests for MDAnalysis during the build. But I've encountered a problem here. Our tests package depends on MDAnalysis installed, but I have to build a conda package of the tests before I can build the MDAnalysis package (Since I want to run the tests for MDAnalsysis when it's build)

The solution for this is easy. I remove the MDAnalysis dependency from the tests package. Then we only warn about the missing MDAnalysis package once we actually run the tests.

How about that?

jbarnoud · 2016-03-04T13:06:07Z

I am not sure I understand. You want to run the test while you are building the conda package? Do you have to? Can't you test the build afterwards as we currently do?

I do not like removing the dependency to MDAnalysis in MDAnalysisTests. One of the reason is just semantics: the tests require MDAnalysis while MDAnalysis works without the tests; the other reason is that the dependency to MDAnalysis also tests that the versions match.

kain88-de · 2016-03-04T13:21:04Z

I am not sure I understand. You want to run the test while you are building the conda package? Do you have to? Can't you test the build afterwards as we currently do?

The process of building a conda package allows to test the build in an isolated environment after the package has been created. This means it builds the package and then runs the tests for you if you tell conda how. I think this is very convenient to setup because it means we can never forget to run the tests on a published package.

If I build a conda package that includes the tests as well we should get around this. The disadvantage would be that our conda package becomes about 30-40 MB.

kain88-de · 2016-03-04T13:26:57Z

I'm doing this for the GridDataFormats package. The problem I'm having is that MDAnalysisTests has to be installed from a conda package currently. I haven't found a way around that yet.

richardjgowers · 2016-03-04T13:51:00Z

So for what it's worth, I already manually test in a new env every package that hits pypi, but I guess the conda package is a separate entity to the pypi package, so it isn't a bad thing to test that separately.

I think maybe distribute them together as one package? 40mb isn't too terrible?

kain88-de · 2016-03-04T14:19:10Z

I think maybe distribute them together as one package? 40mb isn't too terrible?

depending how many people use the wifi during a winter/summer school. But there we could also distribute USB-sticks with the compiled packages (or even complete conda installation?).

hainm · 2016-03-04T17:08:13Z

@kain88-de I don't think it's necessary to run full test when building conda. You pull code from github, which is already tested on travis. I think you only need to run small tests to make sure that the conda build correctly.

kain88-de · 2016-03-06T13:18:16Z

I think you only need to run small tests to make sure that the conda build correctly

Yes and the best for this is to run the full test-suite. Because only then do we run all imports for the build modules guaranteed..

richardjgowers · 2016-03-06T13:24:41Z

You could raise an Issue to shrink down the test data tbh. There's a trr which is 10mb alone and 10 frames, surely we can shrink this to 2-3 frames (and test iteration with a 10 atom system of 1000 frames if need be). We could probably half the size fairly easy with no real loss of tests.

kain88-de · 2016-03-06T13:48:32Z

This is the most recent version. I'm using the 0.14.0 tag from git. But when I run all tests for some reason `test_transformations.py:test_shear_from_matrix' fails.

richardjgowers · 2016-03-06T14:11:55Z

Hmm, I can't remember if that actually uses a np.matrix, maybe something to do with that changed between versions of np?

kain88-de · 2016-03-06T14:24:29Z

strange thing is this seems to fail on develop as well. I'm looking into it with a clean conda environment and checkout of the git repository

kain88-de · 2016-03-07T10:03:32Z

OK I have a new version uploaded onto my anaconda account. Unfortunately I can't produce osx builds because I'm not on an osx system. I will look into the anaconda could build platform to generate packages for osx as well.

kain88-de · 2016-03-08T07:50:32Z

So the OSX build has to wait a little bit. I can't use the anaconda cloud because they only offer linux-64 architecture for builds. But once this is merged I can do a osx-64 build at work.

Otherwise I've uploaded a mdanalysis build to the official channel

It would be nice if someone else could try out the package before I merge this and make an official blog post.

conda config --add channels MDAnalysis
conda install mdanalysis

hainm · 2016-03-08T07:52:44Z

@kain88-de you can use travis of osx.

hainm · 2016-03-08T07:53:33Z

matrix:
  include:
    - { os: linux, env: PYTHON_VERSION=2.7 }
    - { os: linux, env: PYTHON_VERSION=3.4 }
    - { os: linux, env: PYTHON_VERSION=3.5 }
    - { os: osx, env: PYTHON_VERSION=3.4 }

hainm · 2016-03-08T07:54:24Z

anh no build for py3 yet?

jbarnoud · 2016-03-08T07:55:24Z

MDAnalysis is not yet fully functional on py3. And clearly not tested
enough.

On 08-03-16 08:54, Hai Nguyen wrote:

anh no build for py3 yet?

—
Reply to this email directly or view it on GitHub
#751 (comment).

kain88-de · 2016-03-08T08:08:08Z

@kain88-de you can use travis of osx.

I know that travis supports osx. But currently I don't want to spend time figuring out how to connect it with conda and have the right things build and published for the correct branches. That is something for later.

kain88-de · 2016-03-09T16:17:16Z

Did anyone test the linux conda build yet?

kain88-de · 2016-03-16T06:26:56Z

I'll try to make a osx build today. Shouldn't be to hard now. If that works fine I'd say we are good to go.

Scripts to build tempdir dependency conda package and a combined MDAnalysis and MDAnalysistests package. [skip ci]

[skip ci]

kain88-de · 2016-03-16T09:49:46Z

OK the OSX build isn't working straight forward as I'd hoped. For some reason nose can't find the test-modules. Since the linux module is working we can also announce it officially.

kain88-de · 2016-03-16T15:52:29Z

may be add 1 step-install too:

btw that is also accessible as information by clicking on the anaconda badge I have added to the readme.rst. And I wonder that it works because it didn't work for me when I tried.

orbeckst · 2016-03-17T00:38:46Z

maintainer/conda/MDAnalysis/build.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+MDA_USE_OPENMP=FALSE pip install package/


Why build the serial version only?

Conda doesn't ship openmp libraries. To avoid that a release package is build with globally installed openmp versions we default to only serial builds on conda do ensure that the binary package truly runs anywhere.

Ok, we should document and mention this. It's actually a good reason for installing from source.

Oliver Beckstein
email: orbeckst@gmail.com

Am Mar 17, 2016 um 1:19 schrieb kain88-de notifications@github.com:

In maintainer/conda/MDAnalysis/build.sh:

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+MDA_USE_OPENMP=FALSE pip install package/
Conda doesn't ship openmp libraries. To avoid that a release package is build with globally installed openmp versions we default to only serial builds on conda do ensure that the binary package truly runs anywhere.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

I am curious: @hainm , do you not have this issue (can't ship OpenMP-enabled binaries) with pytraj or do you not use OpenMP code?

orbeckst · 2016-03-17T00:47:35Z

I think we also want to add installation through conda to the Quick Start Installation.

kain88-de · 2016-03-17T08:20:20Z

I think we also want to add installation through conda to the Quick Start Installation.

I see you opened a issue on the website repo for that. I will take care of these things when this is merged.

[skip ci]

orbeckst · 2016-03-17T16:48:42Z

Thanks.

Oliver Beckstein
email: orbeckst@gmail.com

Am Mar 17, 2016 um 1:20 schrieb kain88-de notifications@github.com:

I think we also want to add installation through conda to the Quick Start Installation.

I see you opened a issue on the website repo for that. I will take care of these things when this is merged.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

richardjgowers · 2016-03-17T18:11:45Z

@orbeckst I've not been using that script, I get the following:

~/code/mdanalysis/maintainer$ ./change_release.sh 0.16.0
./change_release.sh: line 36: ERROR:: command not found
./change_release.sh: line 39: ERROR:: command not found
./change_release.sh: line 42: ERROR:: command not found
Using sed = /bin/sed -E
Setting RELEASE/__version__ in MDAnalysis to 0.16.0
fatal: ambiguous argument 'package/setup.py': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument 'package/setup.py': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
On branch develop
Your branch is up-to-date with 'origin/develop'.

nothing to commit, working directory clean

It looks like the git commands at the bottom aren't working? (My git is 1.9.1, I think that's old by the looks of it)

Also change the versioning in the conda install files.

kain88-de · 2016-03-17T18:41:20Z

The script works semi OK for me. I'll upload a chance soon. But the script can't remove the dev version end so it isn't very useful IHMO.

kain88-de · 2016-03-17T18:59:53Z

@richardjgowers the script is intended to be called from the root of the repository. maintainer/change_release.sh

Otherwise I have updated the change-release script.

richardjgowers · 2016-03-17T19:03:26Z

Derp, thanks. It works now, except for not getting rid of the -dev0 suffix... which is kind of its main use anyway? Maybe make an easy issue for this? (even a python rewrite?)

kain88-de · 2016-03-17T19:04:41Z

Maybe make an easy issue for this? (even a python rewrite?)

sounds good.

kain88-de · 2016-03-18T08:11:06Z

If there are no further comments I think we can merge this and prepare the official announcements.

richardjgowers · 2016-03-18T08:30:20Z

Looks good to me. What will be the first conda release? .14 or .15?

kain88-de · 2016-03-18T09:54:09Z

0.14.0 the one I have currently uploaded.

orbeckst · 2016-03-18T15:25:03Z

Please merge.

Oliver Beckstein
email: orbeckst@gmail.com

Am Mar 18, 2016 um 1:11 schrieb kain88-de notifications@github.com:

If there are no further comments I think we can merge this and prepare the official announcements.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

conda build scripts (fixes #608)

kain88-de mentioned this pull request Mar 1, 2016

conda build #571

Closed

orbeckst mentioned this pull request Mar 1, 2016

MDAnalysis conda build #608

Closed

kain88-de force-pushed the conda branch 2 times, most recently from b883a6f to f6ed69b Compare March 6, 2016 13:46

kain88-de force-pushed the conda branch 2 times, most recently from 18a8275 to f0f4fa0 Compare March 8, 2016 07:34

kain88-de added this to the 0.15.0 milestone Mar 8, 2016

kain88-de added enhancement installation release labels Mar 8, 2016

kain88-de force-pushed the conda branch from 7afdd8a to 9121a02 Compare March 16, 2016 06:25

kain88-de added 2 commits March 16, 2016 08:08

conda build scripts

e068ce6

Scripts to build tempdir dependency conda package and a combined MDAnalysis and MDAnalysistests package. [skip ci]

Add anaconda badge

9927b6a

[skip ci]

kain88-de force-pushed the conda branch from 9121a02 to 8b50fc0 Compare March 16, 2016 08:39

orbeckst reviewed Mar 17, 2016
View reviewed changes

orbeckst mentioned this pull request Mar 17, 2016

add conda installation instructions MDAnalysis/MDAnalysis.github.io#20

Closed

2 tasks

Add changelog

778b5a5

[skip ci]

kain88-de force-pushed the conda branch from 8b50fc0 to 778b5a5 Compare March 17, 2016 08:21

update release script.

abf7aa7

Also change the versioning in the conda install files.

kain88-de added a commit that referenced this pull request Mar 18, 2016

Merge pull request #751 from kain88-de/conda

e3a345d

conda build scripts (fixes #608)

kain88-de merged commit e3a345d into MDAnalysis:develop Mar 18, 2016

tylerjereddy mentioned this pull request Mar 22, 2016

conda build for OS X #795

Closed

kain88-de deleted the conda branch August 23, 2016 14:04

		@@ -0,0 +1,4 @@
		#!/bin/bash

		MDA_USE_OPENMP=FALSE pip install package/

conda build scripts (fixes #608) #751

conda build scripts (fixes #608) #751

Conversation

kain88-de commented Mar 1, 2016

PR Checklist

- [ ] publish blog post detailing how MDAnalysis can be installed from conda

- [ ] add instructions to Quick Start Installation

kain88-de commented Mar 3, 2016

jbarnoud commented Mar 4, 2016

kain88-de commented Mar 4, 2016

kain88-de commented Mar 4, 2016

richardjgowers commented Mar 4, 2016

kain88-de commented Mar 4, 2016

hainm commented Mar 4, 2016

kain88-de commented Mar 6, 2016

richardjgowers commented Mar 6, 2016

kain88-de commented Mar 6, 2016

richardjgowers commented Mar 6, 2016

kain88-de commented Mar 6, 2016

kain88-de commented Mar 7, 2016

kain88-de commented Mar 8, 2016

hainm commented Mar 8, 2016

hainm commented Mar 8, 2016

hainm commented Mar 8, 2016

jbarnoud commented Mar 8, 2016

kain88-de commented Mar 8, 2016

kain88-de commented Mar 9, 2016

kain88-de commented Mar 16, 2016

kain88-de commented Mar 16, 2016

kain88-de commented Mar 16, 2016

orbeckst Mar 17, 2016

Choose a reason for hiding this comment

kain88-de Mar 17, 2016

Choose a reason for hiding this comment

orbeckst Mar 17, 2016

Choose a reason for hiding this comment

orbeckst Mar 17, 2016

Choose a reason for hiding this comment

orbeckst commented Mar 17, 2016

kain88-de commented Mar 17, 2016

orbeckst commented Mar 17, 2016

richardjgowers commented Mar 17, 2016

kain88-de commented Mar 17, 2016

kain88-de commented Mar 17, 2016

richardjgowers commented Mar 17, 2016

kain88-de commented Mar 17, 2016

kain88-de commented Mar 18, 2016

richardjgowers commented Mar 18, 2016

kain88-de commented Mar 18, 2016

orbeckst commented Mar 18, 2016