fatal error Invalidating the module cache with 2.8

[brettbode]

Hi,
I am hitting a Python Error and traceback at the end of creating a module with EasyBuild 2.8. I retested and 2.7 doesn't hit the issue so it appears to be new in 2.8. The package and module seem to build fine and indeed it doesn't appear to depend on what software you are actually building as the error happens after the module is created. This is on a Cray XE system with the traditional modules install. Below is the end of the debug log. The full debug log is attached and the traceback is below. I can get past it by commenting out the invalidate cache code, which occasionally means the cache is out of date, but otherwise works. However, I am not sure how to fix it properly.

Thanks,
Brett

== 2016-05-20 13:02:18,852 build_log.py:152 ERROR Traceback (most recent call last):
File "/mnt/abc/u/staff/bbode/EasyBuild-2.8/software/EasyBuild/2.8.0/lib64/python2.6/site-packages/easybuild_framework-2.8.0-py2.6.egg/easybuild/main.py", line 114, in build_and_install_software
(ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
File "/u/staff/bbode/EasyBuild-2.8/software/EasyBuild/2.8.0/lib64/python2.6/site-packages/easybuild_framework-2.8.0-py2.6.egg/easybuild/framework/easyblock.py", line 2358, in build_and_install_one
result = app.run_all_steps(run_test_cases=run_test_cases)
File "/u/staff/bbode/EasyBuild-2.8/software/EasyBuild/2.8.0/lib64/python2.6/site-packages/easybuild_framework-2.8.0-py2.6.egg/easybuild/framework/easyblock.py", line 2274, in run_all_steps
self.run_step(step_name, step_methods)
File "/u/staff/bbode/EasyBuild-2.8/software/EasyBuild/2.8.0/lib64/python2.6/site-packages/easybuild_framework-2.8.0-py2.6.egg/easybuild/framework/easyblock.py", line 2153, in run_step
step_method(self)()
File "/u/staff/bbode/EasyBuild-2.8/software/EasyBuild/2.8.0/lib64/python2.6/site-packages/easybuild_framework-2.8.0-py2.6.egg/easybuild/framework/easyblock.py", line 2005, in make_module_step
invalidate_module_caches_for(path)
File "/u/staff/bbode/EasyBuild-2.8/software/EasyBuild/2.8.0/lib64/python2.6/site-packages/easybuild_framework-2.8.0-py2.6.egg/easybuild/tools/modules.py", line 1150, in invalidate_module_caches_for
key, subcmd, path, cache[key])
KeyError: ('MODULEPATH=/u/staff/bbode/EasyBuild-2.8/modules/all:/u/staff/bbode/EasyBuild/modules/all/:/sw/bw/modulefiles:/sw/xe/modulefiles:/usr/local/modulefiles:/opt/cray/craype/2.5.4/modulefiles:/opt/cray/gem/modulefiles:/opt/cray/modulefiles:/opt/modulefiles', 'modulecmd', '')
(at easybuild/main.py:147 in build_and_install_software)

easybuild-naHPSm.txt

[boegel]

Thank you for the detailed bug report @brettbode!

The caching mechanism is indeed new in EasyBuild 2.8, it was introduced via #1742 to speed up things.

I think the problem is that your $MODULEPATH includes two entries that point to the same location on the filesystem, probably /EasyBuild-2.8/modules/all and /EasyBuild/modules/all.

This setup is not taken into account by the invalidate_module_caches_for function, making it delete a cache entry twice, which results in the traceback you're seeing.vof course, that's a bug, it shouldn't be trying to delete a cache entry that was just evicted again...

I think the fix is pretty straightforward: just add a 'break' statement below the 'del' (same indent level), to break out of the innermost loop.

Can you confirm the symlink entry in your $MODULEPATH, and verify the proposed fix?

[boegel]

@brettbode found some time to dive in: the break patch I suggested works, cfr. the pull request in #1776

Since this issue presents itself regardless of what you're installing, I think this bug warrants a EasyBuild v2.8.1 release, even though it requires a particular setup to trigger it...

[brettbode]

Yes that does indeed fix the issue! My EasyBuild directory is a symlink to one with a specific version as you note. I am not sure I understand why both are getting onto the MODULEPATH as I am only "using" the symlink, but that does seem to do the trick.

Thanks,
Brett

[boegel]

@brettbode EasyBuild prepends $MODULEPATH with the location of where it will install modules, to ensure that module commands it runs also pick up on modules installed by EasyBuild

It seems that you have /u/staff/bbode/EasyBuild/modules/all in your $MODULEPATH already, and then EasyBuild itself prepends /u/staff/bbode/EasyBuild-2.8/modules/all (via check_module_path, cfr. https://github.com/hpcugent/easybuild-framework/blob/master/easybuild/tools/modules.py#L365).

It does this without checking whether the first entry in $MODULEPATH is already the one it wants; or actually, it does, but it does a hard string comparison ('/u/staff/bbode/EasyBuild-2.8/modules/all' != '/u/staff/bbode/EasyBuild/modules/all').

It probably make sense to enhance that check to do a proper path comparison instead, taking into account possible symlinks. The only possible complication there is that the path must exist, but that should be fine by that time.

[boegel]

PR #1777 fixes the issue when prepending a symlinked path to $MODULEPATH

[boegel]

Both PRs #1776 and #1777 are merged, so I think this issue can be closed.

@brettbode: if you agree, please close

[brettbode]

Agreed.