WIP: Adds support for lazy and optional importing of dependencies

[mnmelo]

This addresses issues #577, #1361 and #1159. I have rebased against @orbeckst's branch issue-1159-analysis-deps, not develop, hence the conflicts.

Namely, it provides an easy way to import modules that are only really loaded on accession. This allows one to:

1. delay module loading for potentially heavy dependencies until they're really needed;
2. import optional dependencies without always having to catch potential ImportErrors in order to replace them with meaningful exit messages;
3. concentrate imports at file headers regardless of the needs in points 1. and 2;
4. via 3., reduce repetition when conditional imports are needed in more than one place.

---

Mechanism

My strategy was to adapt the code from the importing module from the PEAK package. On lazy loading a dummy, hollow module class is created and instantiated that triggers a proper module import on accession.

API

Right now two functions are exposed: lazy.import_module and lazy.import_function that emulate the typical uses of import and from import. I'd like feedback on this.
Here's how it now looks in analysis/psa.py, which uses several lazy/optional imports:

# Optional and/or lazily loaded modules
#from scipy import spatial, cluster
#import matplotlib
from MDAnalysis.lib import lazy
spatial = lazy.import_module("scipy.spatial")
cluster = lazy.import_module("scipy.cluster")

matplotlib = lazy.import_module('matplotlib')
plt = lazy.import_module('matplotlib.pyplot')

sns = lazy.import_module('seaborn.apionly')

Also, as it is now lazy modules lack the option to customize an error message, spitting out a generic

"attempted to use a functionality that requires module xx.yy, but it couldn't be loaded. Please install xx and retry."

For this reason I left the netcdf4 optional dependency untouched, because the error message is quite informative installation-wise. But something to add if this PR gets accepted.

Timing

Since I was developing this alongside @orbeckst's take on #1159 (PR #1404) I took the initiative to also make matplotlib and scipy lazy imports.
I just did some crude timing (100 repeats using UNIX's time) and found that on my machine lazy loading of scipy saves a grand total of 5ms when importing the MDAnalysis.analysis.distances submodule (on top of ~250ms taken to load MDAnalysis). This shows that point 1. above might not be that relevant.

Testing

Added tests for both lazy loads and missing dependencies (making good use of @richardjgowers's block_import), but still in nose format. Since this adds stuff to MDAnalysisTests/lib, which @utkbansal already took care of (#1413), it might be best to adapt to pytest directly before merging.

[kain88-de]

This will only load scipy.sparse and nothing else from scipy?

[mnmelo]

This will be equivalent to
import scipy.sparse
in that it results on the 'scipy' name being added to the namespace.
On module access both scipy and scipy.sparse are fully imported.

The level='base' keyword tells the function to return a reference to the base module (scipy). From then on you use scipy.sparse.whatever, just like it had been imported with import.

The difference to the default import_module call (without level='base') is that in the default case the returned reference is to the submodule. In that case you use sparse = import_module('scipy.sparse'), as equivalent to from scipy import sparse.

[jbarnoud]

This is fancy! The only thing that annoys me a little is that you can end up with a calculation failing at the end because something is missing and you were not warn ahead of time.

[jbarnoud]

Could this be a context manager?

[mnmelo]

The context manager already exists, but inside _imp. I'd rather keep it high-level and not use _imp, but I already have to anyway for Python 3 support...
So I guess the answer is yes :)

[mnmelo]

@jbarnoud, yes, that is a possibility. Right now I don't think any of the analyses commits to heavy calculations before using optional dependencies.
I imagine early loading of optionals could become a design guideline for future (heavy) analyses.

[jbarnoud]

This introduces an odd behaviour for a python user. Maybe it should be documented somewhere; at least in the developer page of the wiki.

When should we merge this? It is going to introduce conflicts in a bunch of test files and in .travis.yml.

[mnmelo]

I opened the PR more as a request-for-comments. I think its best to bring it in only after @utkbansal's GSoC effort, no?

As to the impact to the end-user, I'm not so sure. This is internally hidden, and just as before some dependency errors/warnings only show up when executing analysis code, not at import-time.
What use-case do you foresee to be unexpected for users?

OTOH, I should definitely write up something for would-be contributors of code, as this would be a new guideline for using optional dependencies.

[mnmelo]

There's a potentially neat aspect:
because of how import caching works, if I lazy.import_module('scipy') anywhere, later calls to import scipy still get the hollow LazyModule.

This means that we can condense the calls to all those optional/lazy dependencies in a single file (say, lib/lazy.py) and let the rest of the files use regular import scipy calls.

However, a drawback is that functions cannot be preloaded: issuing from scipy.stats import gaussian_kde will immediately trigger the full loading of scipy.stats. Can be compensated for by always using scipy.stats.gaussian_kde, which @orbeckst prefers, but I'm not so sure about it.