feature : module file generation may be disabled via configuration file

[alalazo]

Rationale

Introduce a configuration file for site-specific customization on module files

Modifications :

• introduced a configuration file for module file generators
• added an enable keyword to explicitly enable the generation of the module files (defaults to 'tcl' and 'dotkit')

Example of use :

$ spack config edit modules

Add the following lines in the file to enable only dotkit file generation:

modules::
  enable: ['dotkit']

[tgamblin]

Nice! Can you make it use positive logic? I think it would be easier to understand if the syntax looked like this:

  ```yaml
  modules:
    enable: ['dotkit', 'tcl', 'lmod']
  ```

And I think it would be easy to make this simple for users if etc/spack/modules.yaml was pre-set with useful defaults. See etc/spack/repos.yaml as an example -- it points to the builtin repo, but users can actually disable this if they want to.

[alalazo]

@tgamblin Sure, will do the change.

[tgamblin]

@alalazo: FYI, per the last telcon, I am thinking of adding another layer to the prefs:

1. defaults: spack-maintained in etc/spack/config/defaults
2. site: site and/or repo-maintained in etc/spack/config/site
3. user-level: user-maintained, in ~/.spack/

These would take progressively higher precedence. This would enable us to easily maintain nice default settings in (1) but sites could override them in (2). Currently there is only etc/spack and if you update that locally then pull, you'll typically get a conflict, so the idea is to separate these and allow them to merge cleanly.

Module config would fit nicely into this. Thoughts?

[luigi-calori]

Hope not adding noise:
could this PR be integrated with #498 so that the layered pref could control the module naming?
Would also be possible to set_up site or user templates for module file content?
One use case at our site is that many modules have the same code that log their usage in order to gather usage statistics.

[tgamblin]

@luigi-calori: I like this idea too -- I mentioned it in #552 and I think it is @alalazo's plan.

I don't see why we couldn't add templates, or site-specific boilerplate as well. But maybe we could add that after the module config is in place in a separate PR?

[luigi-calori]

The naming scheme in #498 would be really helpful at our site as it could allow spack generate modules to co-exist with our current in-house modules, so, if possible, it would be good to have both module-related PR merged.
Could the #498 naming scheme be controlled by module config yaml file?
That would really be convenient for our site.

Regarding template, as I' m really a newbie, I would like to make sure the file config syntax ( yaml, am I right?) could support definition of general strings allowing variable expansion for templating.

Currently the content of the module seem fully defined in lib/spack/spack/modules.py

No customization seem allowed, neither from the individual package.py nor by site or user config yaml files.

For example, at our site, virtualgl module define a custom alias for vglrun and include tcl code to inspect available X display.
Is there already a way to add this in spack ?

If a simple subst syntax could be defined such as @{variable_name} then write() and write_header() functions could be replaced by substitutions in the (default/site/user)template

then the custom tcl instruction of my virtualgl module could be define in it's package like the description and expanded in the template as @{module_footer}

Agree better to have a separate PR, likely when this is merged.
Let me know if worthwhile me to help and how

[alalazo]

@luigi-calori

No customization seem allowed, neither from the individual package.py nor by site or user config yaml files.

#552 has a quite self-explanatory title though (not to mention a description that took me a lot of time to write) 😄

[alalazo]

@tgamblin In fact you're right. My plan on module files generation is more or less:

1. polish this small PR and make it ready to be merged, so that
2. merging lmod : support for hierarchical modules #107 as an experimental feature will be possible (lua module files not generated by default?)

At that point we'll have a central place to customize module files generation behavior and we could proceed with :

• custom naming scheme : at first glance I would say that Modules in directories and fixes for setup-env.csh #498 is a good default for tcl. What I would like to do on top of that is to easily change that default from modules.yaml. I think I'll need to discuss with @glennpj when at this point to coordinate on what to do
• customization of entries from configuration files : filter out not wanted defaults, add variables to modules etc.

[luigi-calori]

Sorry, either I' ve been misunderstood or I was too dumb in reading docs:

I greatly appreciated your PR #552 but maybe I 've missed some feature:

I understood it as solid basis for environment handling.
It let package.py customize it's corresponding module by letting packages define environment for themselves and for their dependencies.

What I did not spot was the ability to add further instructions to modules apart from env variables and paths.

(Tcl) modules can contain arbitrary tcl code such as the tcl code added for module help

I did not spot any way to add arbitrary tcl code to the module generated without changing spack core, that is the reason for my comments and proposal.

Please let me know if it is already possible or if it is considered undesirable.

In the latter case, those (rare) cases where a greater customization is needed should be hand hacked

[glennpj]

@alalazo The #498 PR essentially collects module files of the various versions and variants of a program into a common directory, named for that program. It is useful for both TCL modules and Lmod modules, whether using a flat or hierarchical structure. The naming scheme probably needs to be tweaked but I did not try to tackle that in that PR. :)

[alalazo]

@glennpj by naming scheme I really mean the layout of the module files on disk. What you have done in #498 seems to me quite better than the current state of things. For lmod in #107 I have done basically the same, implementing the usual "Core/Compiler/MPI" subdivision.

The idea of the customization is just to permit schemes like e.g.:

• <compiler>-<compiler-version>/<package>/<version>-<hash>

and all sorts of similar combinations that the various sites would like to have instead of the default ones.

[alalazo]

@tgamblin updated description and made the logic positive.

[tgamblin]

@alalazo: looks good to me -- ready to merge?

[alalazo]

@tgamblin yes

[tgamblin]

Thanks!