add support for alternative module naming schemes

[boegel]

This PR is far from finished, but intends to solve #173 when it's done.

Comments/suggestions w.r.t. the current draft, which works and even includes a unit test to check the functionality, are welcome.

I will keep updating this when I find time, and will request a full (re)review when I consider it ready for merging.

[fgeorgatos]

Hi Kenneth,

nice work.

I think that this is actually VERY close to the finishing line, since we'd like to see just 2 cases covered:
a) the equivalent of .lower() applied on the module namespace
b) a generalization via a mapping file (containing two columns: original name & mapped name) that allows to mangle the naming scheme to the will of each site.

I think that meeting these two targets would be sufficient to consider original #173 case closed; other schemes I'd consider them separate feature requests! (including the support for nested names, like openfoam/2.1/gnu etc)

[boegel]

It is certainly a big step towards #173, but it's not finished yet:

• the current 'magic' import of det_full_module_name from the module_naming_scheme module needs to be done in another way, probably looking for (a subclass) of a ModuleNamingScheme class, together with a check to see how EasyBuild was configured; we need to make it stricter, not just a magic import that might sometimes work and lead to surprises (e.g. mixed naming schemes being used :-/)
• some kind of --test-module-naming-scheme command line option should be present, that performs a sanity check of the current active module naming scheme; we need to be somewhat sure that the naming scheme that will be used makes sense, i.e. that is doesn't simply always return <name>/<version>, and ignore toolchain, versionprefix/suffix, etc; maybe this check should be run every time eb is used to build something (i.e., really enforce 'valid' module naming schemes)...
• currently nothing is there that maps dependencies provided with the EasyBuild naming scheme used for easyconfigs to the custom module naming scheme that is in place; that's required, otherwise building anything that has dependencies would fail because they can't be found...

As a sidenote: there's really nothing special about nested names, so supporting those is free (as far as I can tell right now, maybe I'm missing something).

I hope I can find more time to continue this soon...

[fgeorgatos]

we need to be somewhat sure that the naming scheme that will be used makes sense, i.e. that is doesn't simply always return /, and ignore toolchain, versionprefix/suffix, etc; maybe this check should be run every time eb is used to build something (i.e., really enforce 'valid' module naming schemes)...

I really think the above is extra buggage that is truly optional:

• sites have already made compromises in their naming schemes (eg. doing exactly that: fftw/3.3.3); not giving them the ability to implement it, basically means depriving them of an intermediate step to migrate to EasyBuild!
• there is no perfect naming scheme, no matter what (food for thought: shouldn't pyTables-2.4.0-ictce-4.1.13-Python-2.7.3.eb have -gpfs somewhere in its name?); see: hash functions => information reduction

currently nothing is there that maps dependencies provided with the EasyBuild naming scheme used for easyconfigs to the custom module naming scheme that is in place; that's required, otherwise building anything that has dependencies would fail because they can't be found...

Yes, that is needed and should work for both 2 cases mentioned, to consider this viable.

there's really nothing special about nested names, so supporting those is free

I think it would work well if all subdirs (/gcc/4.8.1) get auto-created, yet it will imply certain troubles if people modify the mapping scheme mid-flight (you probably need to start clean, whenever you play with the naming scheme)

[boegel]

One step closer to our goal with the commits I've just included.

This now enables building a module with a custom module naming scheme, e.g.:

$ export EASYBUILD_MODULE_NAMING_SCHEME=MyModuleNamingScheme

$ eb gzip-1.5-goolf-1.4.10.eb
== temporary log file in case of crash /var/folders/6y/x4gmwgjn5qz63b7ftg4j_40m0000gn/T/easybuild-5I_seD.log
== resolving dependencies ...
== processing EasyBuild easyconfig /Users/kehoste/.local/lib/python2.7/site-packages/easybuild_easyconfigs-1.6.0.0dev-py2.7.egg/easybuild/easyconfigs/g/gzip/gzip-1.5-goolf-1.4.10.eb
== building and installing gnu-openmpi-gzip-1.5...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== packaging...
== postprocessing...
== sanity checking...
== cleaning up...
== creating module...
== COMPLETED: Installation ended successfully
== Results of the build can be found in the log file /var/folders/6y/x4gmwgjn5qz63b7ftg4j_40m0000gn/T/easybuild-gzip-1.5-20130822.165933.log
== Build succeeded for 1 out of 1
== temporary log file /var/folders/6y/x4gmwgjn5qz63b7ftg4j_40m0000gn/T/easybuild-5I_seD.log has been removed.

$ module av gnu/openmpi/gzip/1.5

-------------------------------------------------------------------- /Users/kehoste/.local/easybuild/modules/all --------------------------------------------------------------------
gnu/openmpi/gzip/1.5

$ module show gnu/openmpi/gzip/1.5
-------------------------------------------------------------------
/Users/kehoste/.local/easybuild/modules/all/gnu/openmpi/gzip/1.5:

module-whatis    gzip (GNU zip) is a popular data compression program as a replacement for compress - Homepage: http://www.gnu.org/software/gzip/ 
conflict     gzip 
module       load goolf/1.4.10 
prepend-path     MANPATH /Users/kehoste/.local/easybuild/software/gnu/openmpi/gzip/1.5/share/man 
prepend-path     PATH /Users/kehoste/.local/easybuild/software/gnu/openmpi/gzip/1.5/bin 
setenv       EBROOTGZIP /Users/kehoste/.local/easybuild/software/gnu/openmpi/gzip/1.5 
setenv       EBVERSIONGZIP 1.5 
setenv       EBDEVELGZIP /Users/kehoste/.local/easybuild/software/gnu/openmpi/gzip/1.5/easybuild/gnu-openmpi-gzip-1.5-easybuild-devel 
-------------------------------------------------------------------

Still required: properly handling dependency resolution when a custom module naming scheme is selected.

[boegel]

The last commits take care of a significant refactoring of the modules.py module, to flesh out the hardcoding of the EasyBuild module naming scheme (<name>/<versionprefix>-<version>-<toolchain>-<versionsuffix>).
It still needs some work to get the dependency resolution to use the customized module naming scheme, but the major part should be done already...
This has also been tested with both lmod and modulecmd...

[fgeorgatos]

c-o-n-g-r-a-t-u-l-a-t-i-o-n-s

i'm sure you are really close to finalizing this; on behalf of many HPC sites (and their users), a big applause.
I have some potential naming scheme patterns to test, but this can happen as a follow up activity, after this ticket.

[boegel]

I'm pretty close, yes, but not finished yet. Beyond a single gzip build that didn't require dependency resolving, this hasn't received much testing yet either, so it won't be ready for EB v1.7.
This PR is also huge, and will need some time to be reviewed by someone familiar with the framework, e.g. @JensTimmerman or @stdweird...

[boegel]

Again, one step closer. Dependency resolution in the light of a custom module naming scheme now works:

hpcbunny:TMP kehoste$ module av gnu

-------------------------------------------------------------------- /Users/kehoste/.local/easybuild/modules/all --------------------------------------------------------------------
gnu/openmpi/bzip2/1.0.6     gnu/openmpi/gzip/1.5        gnu/openmpi/libreadline/6.2 gnu/openmpi/mothur/1.30.2   gnu/openmpi/ncurses/5.9
hpcbunny:TMP kehoste$ module show gnu/openmpi/mothur
-------------------------------------------------------------------
/Users/kehoste/.local/easybuild/modules/all/gnu/openmpi/mothur/1.30.2:

module-whatis    Mothur is a single piece of open-source, expandable software to fill the bioinformatics needs of
the microbial ecology community. - Homepage: http://www.mothur.org/ 
conflict     Mothur 
module       load goolf/1.4.10 
module       load gnu/openmpi/libreadline/6.2 
module       load gnu/openmpi/ncurses/5.9 
module       load gnu/openmpi/bzip2/1.0.6 
module       load gnu/openmpi/gzip/1.5 
prepend-path     PATH /Users/kehoste/.local/easybuild/software/gnu/openmpi/mothur/1.30.2/bin 
setenv       EBROOTMOTHUR /Users/kehoste/.local/easybuild/software/gnu/openmpi/mothur/1.30.2 
setenv       EBVERSIONMOTHUR 1.30.2 
setenv       EBDEVELMOTHUR /Users/kehoste/.local/easybuild/software/gnu/openmpi/mothur/1.30.2/easybuild/gnu-openmpi-mothur-1.30.2-easybuild-devel 
-------------------------------------------------------------------

The det_full_module_name function needs to be enhanced further to make sure the dependency resolution also works for module naming schemes that require more than name, version, versionsuffix, toolchain (i.e., what is provided for dependencies in the easyconfig file of the 'parent')...

That shouldn't be too much work on top of this. And then: testing, testing, testing. :)

I'll try and document how to play around with a custom module naming scheme ASAP.

[fgeorgatos]

Hi Ken,

one more step of progress.

although you would be reluctant to do this, IFF dependency resolution is
even partially working,
I strongly suggest to add this in in v1.7 and call it experimental
feature. That's totally fair.

I very well understand that as a developer you may be reluctant to do so;
but speaking for myself and probably many others, the release is basically
one more distribution channel and it is certainly more effective than
"hey guys, git checkout the develop branches across 3 repos, follow X-Y-Z
wiki page and see what happens".

To sum up, even if this is only partially working, please let it slip
through and we'll give you nice feedback.

On Wed, Aug 28, 2013 at 5:31 PM, Kenneth Hoste notifications@github.comwrote:

Again, one step closer. Dependency resolution in the light of a custom
module naming scheme now works:

hpcbunny:TMP kehoste$ module av gnu

-------------------------------------------------------------------- /Users/kehoste/.local/easybuild/modules/all --------------------------------------------------------------------
gnu/openmpi/bzip2/1.0.6 gnu/openmpi/gzip/1.5 gnu/openmpi/libreadline/6.2 gnu/openmpi/mothur/1.30.2 gnu/openmpi/ncurses/5.9

hpcbunny:TMP kehoste$ module show gnu/openmpi/mothur

/Users/kehoste/.local/easybuild/modules/all/gnu/openmpi/mothur/1.30.2:

module-whatis Mothur is a single piece of open-source, expandable software to fill the bioinformatics needs of
the microbial ecology community. - Homepage: http://www.mothur.org/
conflict Mothur
module load goolf/1.4.10
module load gnu/openmpi/libreadline/6.2
module load gnu/openmpi/ncurses/5.9
module load gnu/openmpi/bzip2/1.0.6
module load gnu/openmpi/gzip/1.5
prepend-path PATH /Users/kehoste/.local/easybuild/software/gnu/openmpi/mothur/1.30.2/bin
setenv EBROOTMOTHUR /Users/kehoste/.local/easybuild/software/gnu/openmpi/mothur/1.30.2
setenv EBVERSIONMOTHUR 1.30.2

setenv EBDEVELMOTHUR /Users/kehoste/.local/easybuild/software/gnu/openmpi/mothur/1.30.2/easybuild/gnu-openmpi-mothur-1.30.2-easybuild-devel

The det_full_module_name function needs to be enhanced further to make
sure the dependency resolution also works for module naming schemes that
require more than name, version, versionsuffix, toolchain (i.e., what is
provided for dependencies in the easyconfig file of the 'parent')...

That shouldn't be too much work on top of this. And then: testing,
testing, testing. :)

I'll try and document how to play around with a custom module naming
scheme ASAP.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/679#issuecomment-23423885
.

[boegel]

I might be willing to include it so people can easily experiment with it, but if you look at the diff, you'll see that some major refactoring was required to be able to support this.
I currently have no view on what was broken by doing this huge refactoring effort, and won't have time to test this before the v1.7 release which is due next Monday.
I'd rather release on time (we already skipped a release end of July!), than release 1-2-3 weeks later because of this (nice, yes) feature.

It also needs a thorough review first, as mentioned before. I won't rush this through just to make the v1.7 release when the next release is only a month away (still very much in time for SC'13, which I find very important).

[fgeorgatos]

OK.

I think all that matters here is that you still have the freedom to
decommission the feature, without breaking other things;
if that criterion is met, then anything goes as experimental feature...

On Wed, Aug 28, 2013 at 8:59 PM, Kenneth Hoste notifications@github.comwrote:

I might be willing to include it so people can easily experiment with it,
but if you look at the diff, you'll see that some major refactoring was
required to be able to support this.
I currently have no view on what was broken by doing this huge refactoring
effort, and won't have time to test this before the v1.7 release which is
due next Monday.
I'd rather release on time (we already skipped a release end of July!),
than release 1-2-3 weeks later because of this (nice, yes) feature.

It also needs a thorough review first, as mentioned before. I won't rush
this through just to make the v1.7 release when the next release is only a
month away (still very much in time for SC'13, which I find very important).

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/679#issuecomment-23438823
.

[boegel]

The problem is that it's way more than just an added feature 'on the side'. It required a significant amount of changing current functionality on which everything else supports.
Even worse, I had to break the current API of modules.py (which is OK, since the stuff I touched is considered to be internal to the framework, it shouldn't affect easyblocks), and to get full confidence I didn't break any features/eb command line options, it needs to be thoroughly tested/reviewed, which will take time.
It's not because it looks ready and works for basic toy cases, that it close to be completed...

[stdweird]

die camelCase die

[boegel]

Killed it.

[stdweird]

@boegel i reread it, either i'm very tired or i only have some minor remarks. i'd say it's ready to run the release process with alt naming.

[JensTimmerman]

use self.log.debug("Adding %s as a module dependency", dep_mod_name), it differs the string interpolation to when it is actually needed (when debug is on)
Or is this not working with easybuildlog?
(same for everywhere else)

[boegel]

EasyBuildLog is an instance of FancyLogger, so I don't see why it wouldn't work...

However, this is (way) out of the scope of this PR, which is already huge.

If we do this, we should apply that optimization consistently across the whole codebase.

I suggest you open an issue (or a PR ;-)) for this...

[JensTimmerman]

looks better

[boegel]

Tested again it's current state, a robot build of DOLFIN, OpenFOAM, WPS, scipy with various toolchains (goalf, goolf, ictce) with a custom module naming scheme (i.e. the one currently shown in the documentation on the wiki) works perfectly. Builds with the default module naming have also been verified to still work as expected (and this will be well tested as part of the regression test anyway).

So, I see no reason to hold this back any further, and I'm merging it in, well in time for EasyBuild v1.8.0.

This work will be followed up in #687, where support for more 'advanced' module naming schemes will be implemented, together with some more supporting functionality that module naming schemes can use, most likely to be housed in the easybuild.tools.module_naming_scheme.utilities namespace.

Thank to everyone who contributed to this, I consider this a very important feature!