Merge branch 'develop' into fix_mpich_inconsistencies

CONTRIBUTING.md

@@ -138,7 +138,7 @@ You might also want to look into [hub](https://github.com/defunkt/hub) for more
 
 ### Review process
 
-A member of the EasyBuild team will then review your pull request, paying attention to what you're contributing, how you implemented it and [Code style](https://github.com/hpcugent/easybuild/wiki/Code-style).
+A member of the EasyBuild team will then review your pull request, paying attention to what you're contributing, how you implemented it and [code style](http://easybuild.readthedocs.org/en/latest/Code_style.html).
 
 Most likely, some remarks will be made on your pull request. Note that this is nothing personal, we're just trying to keep the EasyBuild codebase as high quality as possible. Even when an EasyBuild team member makes changes, the same public review process is followed.
 

README.rst

@@ -1,54 +1,57 @@
-Build status - *master branch (Python 2.4, Python 2.6, Python 2.7)*
-
-.. image:: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_master-python24/badge/icon
-    :target: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_master-python24/
-.. image:: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_master/badge/icon
-    :target: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_master/  
-.. image:: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_master-python27/badge/icon
-    :target: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_master-python27/ 
-
-Build status - *develop branch (Python 2.4, Python 2.6, Python 2.7)*
-
-.. image:: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_develop-python24/badge/icon
-    :target: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_develop-python24/  
-.. image:: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_develop/badge/icon
-    :target: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_develop/  
-.. image:: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_develop-python27/badge/icon
-    :target: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_develop-python27/
-
-EasyBuild: building software with ease
---------------------------------------
-
-The easybuild-framework package is the basis for EasyBuild
-(http://hpcugent.github.com/easybuild), a software build and
-installation framework written in Python that allows you to install
-software in a structured, repeatable and robust way.
-
-This package contains the EasyBuild framework that supports the
-implementation and use of so-called easyblocks, that implement the
-software install procedure for a particular (group of) software
+.. image:: http://hpcugent.github.io/easybuild/images/easybuild_logo_small.png
+   :align: center
+
+`EasyBuild <https://hpcugent.github.io/easybuild>`_ is a software build
+and installation framework that allows you to manage (scientific) software
+on High Performance Computing (HPC) systems in an efficient way.
+
+The **easybuild-framework** package is the core of EasyBuild. It
+supports the implementation and use of so-called easyblocks which
+implement the software install procedure for a particular (group of) software
 package(s).
 
-The code of the easybuild-framework package is hosted on GitHub, along
+The EasyBuild documentation is available at http://easybuild.readthedocs.org/.
+
+The EasyBuild framework source code is hosted on GitHub, along
 with an issue tracker for bug reports and feature requests, see
 http://github.com/hpcugent/easybuild-framework.
 
-The EasyBuild documentation is available on the GitHub wiki of the
-easybuild meta-package, see
-http://github.com/hpcugent/easybuild/wiki/Home.
-
-Related packages: 
-- easybuild-easyblocks
-(http://pypi.python.org/pypi/easybuild-easyblocks): a collection of
-easyblocks that implement support for building and installing (groups
-of) software packages.
-
-- easybuild-easyconfigs
-(http://pypi.python.org/pypi/easybuild-easyconfigs): a collection of
-example easyconfig files that specify which software to build, and using
-which build options; these easyconfigs will be well tested with the
-latest compatible versions of the easybuild-framework and
-easybuild-easyblocks packages.
-
-The code in the vsc directory originally comes from VSC-tools
-(https://github.com/hpcugent/VSC-tools).
+Related Python packages:
+
+* **easybuild-easyblocks**
+
+  * a collection of easyblocks that implement support for building and installing (groups of) software packages
+  * GitHub repository: http://github.com/hpcugent/easybuild-easyblocks
+  * package on PyPi: https://pypi.python.org/pypi/easybuild-easyblocks
+
+* **easybuild-easyconfigs**
+
+  * a collection of example easyconfig files that specify which software to build,
+    and using which build options; these easyconfigs will be well tested
+    with the latest compatible versions of the easybuild-framework and easybuild-easyblocks packages
+  * GitHub repository: http://github.com/hpcugent/easybuild-easyconfigs
+  * PyPi: https://pypi.python.org/pypi/easybuild-easyconfigs
+
+The code in the ``vsc`` directory originally comes from the *vsc-base* package
+(https://github.com/hpcugent/vsc-base).
+
+
+*Build status overview:*
+
+* **master** branch *(Python 2.4, Python 2.6, Python 2.7)*
+
+  .. image:: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_master-python24/badge/icon
+      :target: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_master-python24/
+  .. image:: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_master/badge/icon
+      :target: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_master/
+  .. image:: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_master-python27/badge/icon
+      :target: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_master-python27/
+
+* **develop** branch *(Python 2.4, Python 2.6, Python 2.7)*
+
+  .. image:: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_develop-python24/badge/icon
+      :target: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_develop-python24/
+  .. image:: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_develop/badge/icon
+      :target: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_develop/
+  .. image:: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_develop-python27/badge/icon
+      :target: https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-framework_unit-test_hpcugent_develop-python27/

easybuild/scripts/bootstrap_eb.py

@@ -431,7 +431,7 @@ def main():
     info('')
     info("By default, EasyBuild will install software to $HOME/.local/easybuild.")
     info("To install software with EasyBuild to %s, make sure $EASYBUILD_INSTALLPATH is set accordingly." % install_path)
-    info("See https://github.com/hpcugent/easybuild/wiki/Configuration for details on configuring EasyBuild.")
+    info("See http://easybuild.readthedocs.org/en/latest/Configuration.html for details on configuring EasyBuild.")
 
 # template easyconfig file for EasyBuild
 EB_EC_FILE = """

easybuild/tools/config.py

@@ -387,12 +387,12 @@ def init(options, config_options_dict):
 
 def init_build_options(build_options=None, cmdline_options=None):
     """Initialize build options."""
-    # building a dependency graph implies force, so that all dependencies are retained
-    # and also skips validation of easyconfigs (e.g. checking os dependencies)
 
     active_build_options = {}
 
     if cmdline_options is not None:
+        # building a dependency graph implies force, so that all dependencies are retained
+        # and also skips validation of easyconfigs (e.g. checking os dependencies)
         retain_all_deps = False
         if cmdline_options.dep_graph:
             _log.info("Enabling force to generate dependency graph.")

easybuild/tools/options.py

@@ -75,18 +75,32 @@ class EasyBuildOptions(GeneralOption):
 
     ALLOPTSMANDATORY = False  # allow more than one argument
 
+    def __init__(self, *args, **kwargs):
+        """Constructor."""
+
+        self.default_robot_paths = get_paths_for(subdir=EASYCONFIGS_PKG_SUBDIR, robot_path=None) or []
+
+        # set up constants to seed into config files parser, by section
+        self.go_cfg_constants = {
+            self.DEFAULTSECT: {
+                'DEFAULT_ROBOT_PATHS': (os.pathsep.join(self.default_robot_paths),
+                                        "List of default robot paths ('%s'-separated)" % os.pathsep),
+            }
+        }
+
+        # update or define go_configfiles_initenv in named arguments to pass to parent constructor
+        go_cfg_initenv = kwargs.setdefault('go_configfiles_initenv', {})
+        for section, constants in self.go_cfg_constants.items():
+            constants = dict([(name, value) for (name, (value, _)) in constants.items()])
+            go_cfg_initenv.setdefault(section, {}).update(constants)
+
+        super(EasyBuildOptions, self).__init__(*args, **kwargs)
+
     def basic_options(self):
         """basic runtime options"""
         all_stops = [x[0] for x in EasyBlock.get_steps()]
         strictness_options = [run.IGNORE, run.WARN, run.ERROR]
 
-        easyconfigs_pkg_paths = get_paths_for(subdir=EASYCONFIGS_PKG_SUBDIR, robot_path=None)
-        if easyconfigs_pkg_paths:
-            default_robot_paths = easyconfigs_pkg_paths
-        else:
-            self.log.warning("basic_options: unable to determine easyconfigs pkg path for --robot-paths default")
-            default_robot_paths = []
-
         descr = ("Basic options", "Basic runtime options for EasyBuild.")
 
         opts = OrderedDict({
@@ -100,7 +114,7 @@ def basic_options(self):
             'robot': ("Enable dependency resolution, using easyconfigs in specified paths",
                       'pathlist', 'store_or_None', [], 'r', {'metavar': 'PATH[%sPATH]' % os.pathsep}),
             'robot-paths': ("Additional paths to consider by robot for easyconfigs (--robot paths get priority)",
-                            'pathlist', 'store', default_robot_paths, {'metavar': 'PATH[%sPATH]' % os.pathsep}),
+                            'pathlist', 'store', self.default_robot_paths, {'metavar': 'PATH[%sPATH]' % os.pathsep}),
             'skip': ("Skip existing software (useful for installing additional packages)",
                      None, 'store_true', False, 'k'),
             'stop': ("Stop the installation after certain step", 'choice', 'store_or_None', 'source', 's', all_stops),
@@ -256,6 +270,8 @@ def informative_options(self):
         descr = ("Informative options", "Obtain information about EasyBuild.")
 
         opts = OrderedDict({
+            'avail-cfgfile-constants': ("Show all constants that can be used in configuration files",
+                                        None, 'store_true', False),
             'avail-easyconfig-constants': ("Show all constants that can be used in easyconfigs",
                                            None, 'store_true', False),
             'avail-easyconfig-licenses': ("Show all license constants that can be used in easyconfigs",
@@ -369,7 +385,7 @@ def postprocess(self):
 
         # prepare for --list/--avail
         if any([self.options.avail_easyconfig_params, self.options.avail_easyconfig_templates,
-                self.options.list_easyblocks, self.options.list_toolchains,
+                self.options.list_easyblocks, self.options.list_toolchains, self.options.avail_cfgfile_constants,
                 self.options.avail_easyconfig_constants, self.options.avail_easyconfig_licenses,
                 self.options.avail_repositories, self.options.show_default_moduleclasses,
                 self.options.avail_modules_tools, self.options.avail_module_naming_schemes,
@@ -423,6 +439,11 @@ def _postprocess_config(self):
     def _postprocess_list_avail(self):
         """Create all the additional info that can be requested (exit at the end)"""
         msg = ''
+
+        # dump supported configuration file constants
+        if self.options.avail_cfgfile_constants:
+            msg += self.avail_cfgfile_constants()
+
         # dump possible easyconfig params
         if self.options.avail_easyconfig_params:
             msg += self.avail_easyconfig_params()
@@ -469,6 +490,23 @@ def _postprocess_list_avail(self):
             print msg
         sys.exit(0)
 
+    def avail_cfgfile_constants(self):
+        """
+        Return overview of constants supported in configuration files.
+        """
+        lines = [
+            "Constants available (only) in configuration files:",
+            "syntax: %(CONSTANT_NAME)s",
+        ]
+        for section in self.go_cfg_constants:
+            lines.append('')
+            if section != self.DEFAULTSECT:
+                section_title = "only in '%s' section:" % section
+                lines.append(section_title)
+            for cst_name, (cst_value, cst_help) in sorted(self.go_cfg_constants[section].items()):
+                lines.append("* %s: %s [value: %s]" % (cst_name, cst_help, cst_value))
+        return '\n'.join(lines)
+
     def avail_easyconfig_params(self):
         """
         Print the available easyconfig parameters, for the given easyblock.

setup.py

@@ -82,9 +82,7 @@ def find_rel_test():
     version = str(VERSION),
     author = "EasyBuild community",
     author_email = "easybuild@lists.ugent.be",
-    description = """EasyBuild is a software installation framework in Python that allows you to \
-install software in a structured and robust way.
-This package contains the EasyBuild framework, which supports the creation of custom easyblocks that \
+    description = """The EasyBuild framework supports the creation of custom easyblocks that \
 implement support for installing particular (groups of) software packages.""",
     license = "GPLv2",
     keywords = "software build building installation installing compilation HPC scientific",
@@ -96,12 +94,7 @@ def find_rel_test():
     data_files = [
         ('easybuild', ["easybuild/easybuild_config.py"]),
     ],
-    long_description = """This package contains the EasyBuild
-framework, which supports the creation of custom easyblocks that
-implement support for installing particular (groups of) software
-packages.
-
-""" + read("README.rst"),
+    long_description = read('README.rst'),
     classifiers = [
         "Development Status :: 5 - Production/Stable",
         "Environment :: Console",

test/framework/config.py

@@ -31,6 +31,7 @@
 import copy
 import os
 import shutil
+import sys
 import tempfile
 from test.framework.utilities import EnhancedTestCase, init_config
 from unittest import TestLoader
@@ -42,7 +43,7 @@
 from easybuild.tools.config import log_file_format, set_tmpdir, BuildOptions, ConfigurationVariables
 from easybuild.tools.config import get_build_log_path, DEFAULT_PATH_SUBDIRS, init_build_options, build_option
 from easybuild.tools.environment import modify_env
-from easybuild.tools.filetools import write_file
+from easybuild.tools.filetools import mkdir, write_file
 from easybuild.tools.repository.filerepo import FileRepository
 from easybuild.tools.repository.repository import init_repository
 
@@ -443,10 +444,22 @@ def test_generaloption_config_file(self):
         self.assertEqual(source_paths(), [os.path.join(os.getenv('HOME'), '.local', 'easybuild', 'sources')])  # default
         self.assertEqual(install_path(), os.path.join(testpath2, 'software'))  # via config file
 
+        # copy test easyconfigs to easybuild/easyconfigs subdirectory of temp directory
+        # to check whether easyconfigs install path is auto-included in robot path
+        tmpdir = tempfile.mkdtemp(prefix='easybuild-easyconfigs-pkg-install-path')
+        mkdir(os.path.join(tmpdir, 'easybuild'), parents=True)
+
+        test_ecs_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'easyconfigs')
+        shutil.copytree(test_ecs_dir, os.path.join(tmpdir, 'easybuild', 'easyconfigs'))
+
+        orig_sys_path = sys.path[:]
+        sys.path.insert(0, tmpdir)  # prepend to give it preference over possible other installed easyconfigs pkgs
+
         # test with config file passed via environment variable
         cfgtxt = '\n'.join([
             '[config]',
             'buildpath = %s' % testpath1,
+            'robot-paths = /tmp/foo:%(DEFAULT_ROBOT_PATHS)s',
         ])
         write_file(config_file, cfgtxt)
 
@@ -460,6 +473,8 @@ def test_generaloption_config_file(self):
         self.assertEqual(install_path(), os.path.join(os.getenv('HOME'), '.local', 'easybuild', 'software'))  # default
         self.assertEqual(source_paths(), [testpath2])  # via command line
         self.assertEqual(build_path(), testpath1)  # via config file
+        self.assertTrue('/tmp/foo' in options.robot_paths)
+        self.assertTrue(os.path.join(tmpdir, 'easybuild', 'easyconfigs') in options.robot_paths)
 
         testpath3 = os.path.join(self.tmpdir, 'testTHREE')
         os.environ['EASYBUILD_SOURCEPATH'] = testpath2
@@ -474,6 +489,7 @@ def test_generaloption_config_file(self):
         self.assertEqual(build_path(), testpath1)  # via config file
 
         del os.environ['EASYBUILD_CONFIGFILES']
+        sys.path[:] = orig_sys_path
 
     def test_set_tmpdir(self):
         """Test set_tmpdir config function."""

test/framework/options.py

@@ -453,6 +453,39 @@ def test_avail_lists(self):
         if os.path.exists(dummylogfn):
             os.remove(dummylogfn)
 
+    def test_avail_cfgfile_constants(self):
+        """Test --avail-cfgfile-constants."""
+        fd, dummylogfn = tempfile.mkstemp(prefix='easybuild-dummy', suffix='.log')
+        os.close(fd)
+
+        # copy test easyconfigs to easybuild/easyconfigs subdirectory of temp directory
+        # to check whether easyconfigs install path is auto-included in robot path
+        tmpdir = tempfile.mkdtemp(prefix='easybuild-easyconfigs-pkg-install-path')
+        mkdir(os.path.join(tmpdir, 'easybuild'), parents=True)
+
+        test_ecs_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'easyconfigs')
+        shutil.copytree(test_ecs_dir, os.path.join(tmpdir, 'easybuild', 'easyconfigs'))
+
+        orig_sys_path = sys.path[:]
+        sys.path.insert(0, tmpdir)  # prepend to give it preference over possible other installed easyconfigs pkgs
+
+        args = [
+            '--avail-cfgfile-constants',
+            '--unittest-file=%s' % self.logfile,
+        ]
+        outtxt = self.eb_main(args, logfile=dummylogfn)
+        cfgfile_constants = {
+            'DEFAULT_ROBOT_PATHS': os.path.join(tmpdir, 'easybuild', 'easyconfigs'),
+        }
+        for cst_name, cst_value in cfgfile_constants.items():
+            cst_regex = re.compile("^\*\s%s:\s.*\s\[value: .*%s.*\]" % (cst_name, cst_value), re.M)
+            tup = (cst_regex.pattern, outtxt)
+            self.assertTrue(cst_regex.search(outtxt), "Pattern '%s' in --avail-cfgfile_constants output: %s" % tup)
+
+        if os.path.exists(dummylogfn):
+            os.remove(dummylogfn)
+        sys.path[:] = orig_sys_path
+
     def test_list_easyblocks(self):
         """Test listing easyblock hierarchy."""
 

vsc/README.md

@@ -1,3 +1,3 @@
 Code from https://github.com/hpcugent/vsc-base
 
-based on 35fee9d3130a6b52bf83993e73d187e9d46c69bc (vsc-base v1.9.9)
+based on 2146be5301da34043adf4646169e5dfec88cd2f5 (vsc-base v1.9.9)