Three test.framework.suite test failures on an Amazon RHEL instance.

[bbssc3rtp]

Hello. I found the easybuild documentation to be not too well adapted to an Amazon RHEL 7 instance. It required running through the tests and identifying a slew of yum and pip packages that needed to be installed.

Now that those have been indentified, I am down to only three testsuite errors. I was wondering if anybody could help me pinpoint the cause.

python -V
Python 2.7.5

type module
module is a function
module ()
{
    eval $($LMOD_CMD bash "$@");
    [ $? = 0 ] && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}

type -f module
-bash: type: module: not found

 module --version
Modules based on Lua: Version 6.3.4  2016-05-11 17:00
    by Robert McLay mclay@tacc.utexas.edu

[ec2-user@ip-172-31-51-65 ~]$ module av EasyBuild

------------------------------------ /tools/general/app/modules/all -------------------------------------
   EasyBuild/2.8.1 (L)

  Where:
   L:  Module is loaded

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

which -a eb
/tools/general/app/software/EasyBuild/2.8.1/bin/eb

eb --version
This is EasyBuild 2.8.1 (framework: 2.8.1, easyblocks: 2.8.1) on host <removed>
==================================================

$ python -m test.framework.suite
INFO: This is (based on) vsc.install.shared_setup 0.10.6

Running tests...
----------------------------------------------------------------------
..................................EE......E.......................(skipping HgRepository test)
......................................................................................................................................................................................................................................................................................................................................................
======================================================================
ERROR [1.062s]: test_include_module_naming_schemes (test.framework.options.CommandLineOptionsTest)
Test --include-module-naming-schemes.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tools/general/app/software/EasyBuild/2.8.1/lib/python2.7/site-packages/easybuild_framework-2.8.1-py2.7.egg/test/framework/options.py", line 1914, in test_include_module_naming_schemes
    logtxt, _= run_cmd("cd %s; eb %s" % (self.test_prefix, ' '.join(args)), simple=False)
  File "/tools/general/app/software/EasyBuild/2.8.1/lib/python2.7/site-packages/easybuild_framework-2.8.1-py2.7.egg/easybuild/tools/run.py", line 149, in run_cmd
    return parse_cmd_output(cmd, stdouterr, ec, simple, log_all, log_ok, regexp)
  File "/tools/general/app/software/EasyBuild/2.8.1/lib/python2.7/site-packages/easybuild_framework-2.8.1-py2.7.egg/easybuild/tools/run.py", line 397, in parse_cmd_output
    raise EasyBuildError('cmd "%s" exited with exitcode %s and output:\n%s', cmd, ec, stdouterr)
EasyBuildError: 'cmd "cd /tmp/eb-c9_8wa/eb-f4A_Vz/eb-BiXsAQ; eb --avail-module-naming-schemes" exited with exitcode 127 and output:\n/bin/bash: eb: command not found\n'

======================================================================
ERROR [1.066s]: test_include_toolchains (test.framework.options.CommandLineOptionsTest)
Test --include-toolchains.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tools/general/app/software/EasyBuild/2.8.1/lib/python2.7/site-packages/easybuild_framework-2.8.1-py2.7.egg/test/framework/options.py", line 1988, in test_include_toolchains
    logtxt, _= run_cmd("cd %s; eb %s" % (self.test_prefix, ' '.join(args)), simple=False)
  File "/tools/general/app/software/EasyBuild/2.8.1/lib/python2.7/site-packages/easybuild_framework-2.8.1-py2.7.egg/easybuild/tools/run.py", line 149, in run_cmd
    return parse_cmd_output(cmd, stdouterr, ec, simple, log_all, log_ok, regexp)
  File "/tools/general/app/software/EasyBuild/2.8.1/lib/python2.7/site-packages/easybuild_framework-2.8.1-py2.7.egg/easybuild/tools/run.py", line 397, in parse_cmd_output
    raise EasyBuildError('cmd "%s" exited with exitcode %s and output:\n%s', cmd, ec, stdouterr)
EasyBuildError: 'cmd "cd /tmp/eb-c9_8wa/eb-f4A_Vz/eb-oMnjse; eb --list-toolchains" exited with exitcode 127 and output:\n/bin/bash: eb: command not found\n'

======================================================================
ERROR [1.604s]: test_new_update_pr (test.framework.options.CommandLineOptionsTest)
Test use of --new-pr (dry run only).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tools/general/app/software/EasyBuild/2.8.1/lib/python2.7/site-packages/easybuild_framework-2.8.1-py2.7.egg/test/framework/options.py", line 2252, in test_new_update_pr
    self.eb_main(args, do_build=True, raise_error=True, testing=False)
  File "/tools/general/app/software/EasyBuild/2.8.1/lib/python2.7/site-packages/easybuild_framework-2.8.1-py2.7.egg/test/framework/utilities.py", line 276, in eb_main
    raise myerr
AttributeError: 'Remote' object has no attribute 'exists'

----------------------------------------------------------------------
Ran 408 tests in 963.660s

FAILED (errors=3)

Generating XML reports...
ERROR: Not all tests were successful.

[boegel]

@bbssc3rtp The first two failing tests are due to a known issue with the tests themselves, see #1685; I implemented a fix for that today, see #1806.

The last failing test is another problem, but it looks like this could be due to an outdated GitPython, see also #1757 .

[bbssc3rtp]

Thank you! A pip upgrade of GitPython now has me down to two failed tests. It seems that by failing these two tests I have actually passed the test! :)

Moving on.. I have read the install and test documents multiple times, and am still confused as to whether I should be running the next two tests and if so, what needs to be done to make them run? The documentation appears to indicate that the bootstrap should have handled it all for me.

$ python -m test.easyblocks.suite
/usr/bin/python: No module named easyblocks

$ python -m test.easyconfigs.suite
/usr/bin/python: No module named easyconfigs

[boegel]

@bbssc3rtp I guess the documentation is a bit misleading here...

The unit tests for easyblocks and easyconfigs are not included in the EasyBuild installation, because they are of little use there.

While running the EasyBuild framework test suite can bring forward issues with your installation or highlight incompatibilities in your setup, the easyblocks/easyconfigs unit tests are only there to ensure that changes being made to the easyblocks/easyconfigs don't introduce 'silly' issues like syntax errors, missing dependencies or patch files, version conflicts in a software dependency graph, etc.

In an EasyBuild release, these kind of problems are pretty much guaranteed not to be there, since the tests are run over and over again through Travis (http://travis-ci.org) every time a change is being made to the central repositories.

Unless you have a development setup of the easyblocks and easyconfigs, where you will have the tests available, there's little point in running the tests for these EasyBuild components...

That being said, I guess it wouldn't hurt to include those tests in the installation too, especially since it should be fairly trivial to do (some minor changes in the respective setup.py scripts).