Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for --update-lmod-caches (disabled by default) and --compile-lmod-caches (WIP) #1177

Open
wants to merge 11 commits into
base: develop
Choose a base branch
from
Open

add support for --update-lmod-caches (disabled by default) and --compile-lmod-caches (WIP) #1177

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
47c01fa
add support for --update-lmod-cache configure option, disabled by def…
boegel Feb 12, 2015
513917a
only update modules tool after generating final module file
boegel Feb 12, 2015
dd0a659
backup existing cache file to .old.lua, write to lua.new and then ren…
boegel Feb 12, 2015
e65357b
add support for compiling Lmod cache files after updating them
boegel Feb 12, 2015
d259ddc
fix remarks, drop --compile-lmod-cache config option (always compile …
boegel Feb 14, 2015
cf6b6b8
fix broken test
boegel Feb 14, 2015
ed57380
add support for specifying location of Lmod cache timestamp file, cor…
boegel Feb 15, 2015
ae673a0
disable debug output for 'module avail', since it screws up parsing t…
boegel Feb 17, 2015
8d9fa5b
fix typo in modulestool unit test
boegel Feb 18, 2015
476b824
fix using LMOD_USER_CACHE_DIR as default cache directory to update ca…
boegel Feb 18, 2015
099923a
Merge branch 'develop' into compile_all_lmod_cache_things
boegel Feb 23, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion easybuild/framework/easyblock.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1652,7 +1652,10 @@ def make_module_step(self, fake=False):

self.log.info("Module file %s written" % self.module_generator.filename)

self.modules_tool.update()
# only update after generating final module file
if not fake:
self.modules_tool.update()

self.module_generator.create_symlinks()

if not fake:
Expand Down
2 changes: 2 additions & 0 deletions easybuild/tools/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ def mk_full_default_path(name, prefix=DEFAULT_PREFIX):
BUILD_OPTIONS_CMDLINE = {
None: [
'aggregate_regtest',
'compile_lmod_caches',
'download_timeout',
'dump_test_report',
'easyblock',
Expand All @@ -98,6 +99,7 @@ def mk_full_default_path(name, prefix=DEFAULT_PREFIX):
'test_report_env_filter',
'testoutput',
'umask',
'update_lmod_caches',
],
False: [
'allow_modules_tool_mismatch',
Expand Down
13 changes: 13 additions & 0 deletions easybuild/tools/filetools.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,19 @@ def hexdigest(self):
return '0x%s' % (self.checksum & 0xffffffff)


def copy_file(src, dest):
"""
Copy file at specified path to new location.
@param src: source path
@param dest: destination path (full path, incl. filename)
"""
try:
mkdir(os.path.dirname(dest))
shutil.copy2(src, dest)
except OSError, err:
_log.error("Failed to copy %s to %s: %s")


def read_file(path, log_error=True):
"""Read contents of file at given path, in a robust way."""
f = None
Expand Down
72 changes: 48 additions & 24 deletions easybuild/tools/modules.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,12 +48,15 @@
from easybuild.tools.build_log import EasyBuildError
from easybuild.tools.config import build_option, get_modules_tool, install_path
from easybuild.tools.environment import restore_env
from easybuild.tools.filetools import convert_name, mkdir, read_file, path_matches, which
from easybuild.tools.filetools import convert_name, copy_file, mkdir, read_file, path_matches, which, write_file
from easybuild.tools.module_naming_scheme import DEVEL_MODULE_SUFFIX
from easybuild.tools.run import run_cmd
from easybuild.tools.toolchain import DUMMY_TOOLCHAIN_NAME, DUMMY_TOOLCHAIN_VERSION
from vsc.utils.missing import nub

# relative to user's home directory
LMOD_USER_CACHE_RELDIR = '.lmod.d/.cache'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

  --with-useDotFiles=yes/no
                          If yes use ~/.lmod.d/.cache, if no use
                          ~/.lmod.d/__cache__

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is just a default, --update-lmod-caches takes an argument if you want to update caches in another location

since .lmod.d/.cache is the default in Lmod, I think just keeping this as is makes sense


# software root/version environment variable name prefixes
ROOT_ENV_VAR_NAME_PREFIX = "EBROOT"
VERSION_ENV_VAR_NAME_PREFIX = "EBVERSION"
Expand Down Expand Up @@ -831,33 +834,54 @@ def available(self, mod_name=None):

return correct_real_mods

def compile_cache(self, cache_fp):
"""Compile Lmod cache file using luac."""
if build_option('compile_lmod_caches'):
# FIXME: ask Lmod to compile, to ensure that luac that matches the lua used by Lmod is used?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 on letting lmod compile the cache. then we also don't need to test if luac is in PATH etc etc.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's just a side-effect, but yes, that's good

it's important that Lmod uses the luac that matches the lua it's using, since the Lua version has to be part of the filename of the compiled cache (I still need to fix that here)

run_cmd("luac %s" % cache_fp)

def update(self):
"""Update after new modules were added."""
spider_cmd = os.path.join(os.path.dirname(self.cmd), 'spider')
cmd = [spider_cmd, '-o', 'moduleT', os.environ['MODULEPATH']]
self.log.debug("Running command '%s'..." % ' '.join(cmd))
"""Update Lmod cache after new modules were added."""
cache_dir = build_option('update_lmod_caches')

proc = subprocess.Popen(cmd, stdout=PIPE, stderr=PIPE, env=os.environ)
(stdout, stderr) = proc.communicate()
if cache_dir is not None:

if stderr:
self.log.error("An error occured when running '%s': %s" % (' '.join(cmd), stderr))
for cache_type in ['moduleT', 'dbT', 'reverseMapT']:

if self.testing:
# don't actually update local cache when testing, just return the cache contents
return stdout
else:
try:
cache_filefn = os.path.join(os.path.expanduser('~'), '.lmod.d', '.cache', 'moduleT.lua')
self.log.debug("Updating Lmod spider cache %s with output from '%s'" % (cache_filefn, ' '.join(cmd)))
cache_dir = os.path.dirname(cache_filefn)
if not os.path.exists(cache_dir):
mkdir(cache_dir, parents=True)
cache_file = open(cache_filefn, 'w')
cache_file.write(stdout)
cache_file.close()
except (IOError, OSError), err:
self.log.error("Failed to update Lmod spider cache %s: %s" % (cache_filefn, err))
# run spider to create cache contents
spider_cmd = os.path.join(os.path.dirname(self.cmd), 'spider')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not use https://github.com/TACC/Lmod/blob/master/contrib/BuildSystemCacheFile/createSystemCacheFile.sh? isn't it part of a lmod distribution?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it gets installed as a part of Lmod, and it needs to be tweaked anyway, it has hardcoded stuff in it specific to TACC.

I'm just redoing what that script implements, with proper logging (coming up) and error handling in Python, etc.

cmd = [spider_cmd, '-o', cache_type, os.environ['MODULEPATH']]
self.log.debug("Running command '%s'..." % ' '.join(cmd))

# we can't use run_cmd, since we need to separate stdout from stderr
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's so special about stderr? there's no proper exitcode?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spider acts like lmod and modulecmd, I think: the cache contents to stdout, any messages (also non-error) to stderr

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we then extend run_cmd for this use case?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should start using the run commands that are provided by vsc.utils.run, and the function in there should be enhanced to allow spltiting stdout/stderr

that's a long-term effort though, since either we redefine run_cmd to use those, or we drop/deprecate run_cmd all together

proc = subprocess.Popen(cmd, stdout=PIPE, stderr=PIPE, env=os.environ)
(cache_txt, stderr) = proc.communicate()
if stderr:
self.log.error("An error occured when running '%s': %s" % (' '.join(cmd), stderr))

if self.testing:
# don't actually update local cache when testing, just return the cache contents
return cache_txt
else:
cache_fp = os.path.join(cache_dir, '%s.lua' % cache_type)
self.log.debug("Updating Lmod spider cache %s with output from '%s'" % (cache_fp, ' '.join(cmd)))

# back existing cache by copying it to <cache>T.old.lua
# Lmod also considers this filename for the cache in case <cache>T.lua isn't found
if os.path.exists(cache_fp):
old_cache_fp = '%s.old.lua' % os.path.splitext(cache_fp)[0]
copy_file(cache_fp, old_cache_fp)
self.compile_cache(old_cache_fp)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why recompile and not move the old cache (if available)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lmod also picks up moduleT.old.luac_<luaver> and moduleT.old.lua, if they're there, to cover the rare case where moduleT.lua is not in place yet, but the previous version is still around

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, yes, but you're right, I should just move the old cache, rather than recompile is, just realised what you intended to point out


# FIXME: also support compiling cache
# make update as atomic as possible: write cache file under temporary name, and then rename
new_cache_fp = '%s.new' % cache_fp
write_file(new_cache_fp, cache_txt)
try:
os.rename(new_cache_fp, cache_fp)
except OSError, err:
self.log.error("Failed to rename Lmod cache %s to %s: %s" % (new_cache_fp, cache_fp, err))
self.compile_cache(cache_fp)

def prepend_module_path(self, path):
# Lmod pushes a path to the front on 'module use'
Expand Down
13 changes: 11 additions & 2 deletions easybuild/tools/options.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@
from easybuild.tools.modules import avail_modules_tools
from easybuild.tools.module_naming_scheme import GENERAL_CLASS
from easybuild.tools.module_naming_scheme.utilities import avail_module_naming_schemes
from easybuild.tools.modules import LMOD_USER_CACHE_RELDIR, Lmod
from easybuild.tools.ordereddict import OrderedDict
from easybuild.tools.toolchain.utilities import search_toolchain
from easybuild.tools.repository.repository import avail_repositories
Expand Down Expand Up @@ -177,6 +178,7 @@ def override_options(self):
'allow-modules-tool-mismatch': ("Allow mismatch of modules tool and definition of 'module' function",
None, 'store_true', False),
'cleanup-builddir': ("Cleanup build dir after successful installation.", None, 'store_true', True),
'compile-lmod-caches': ("Compile Lmod cache files with luac after updating them", None, 'store_true', False),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does this do? should be just a toggle, that if you run --update-cache also compiles it (but i don't like the idea. lmod should do what is best, i.e. compile if luac is avail).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spider just dumps the cache to stdout, there is no tool included in Lmod (yet?) that takes care of the .old, compiling, etc. I'm making the compilation optional here, since luac may not be around, and for older Lmod version the compiled cache is useless (since Lmod doesn't pick it up).

'deprecated': ("Run pretending to be (future) version, to test removal of deprecated code.",
None, 'store', None),
'download_timeout': ("Timeout for initiating downloads (in seconds)", None, 'store', None),
Expand All @@ -197,10 +199,12 @@ def override_options(self):
'set-gid-bit': ("Set group ID bit on newly created directories", None, 'store_true', False),
'sticky-bit': ("Set sticky bit on newly created directories", None, 'store_true', False),
'skip-test-cases': ("Skip running test cases", None, 'store_true', False, 't'),
'umask': ("umask to use (e.g. '022'); non-user write permissions on install directories are removed",
None, 'store', None),
'optarch': ("Set architecture optimization, overriding native architecture optimizations",
None, 'store', None),
'umask': ("umask to use (e.g. '022'); non-user write permissions on install directories are removed",
None, 'store', None),
'update-lmod-caches': ("Update Lmod cache files in specified directory", None, 'store_or_None',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be module tool flavour neutral.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is no caching in Tcl(/C) env mods... suggestion for a better name?

at least that makes it clear from the name of the command line option that this is specific to Lmod

os.path.join(os.path.expanduser('~'), LMOD_USER_CACHE_RELDIR), {'metavar': 'DIR'}),
})

self.log.debug("override_options: descr %s opts %s" % (descr, opts))
Expand Down Expand Up @@ -405,6 +409,11 @@ def postprocess(self):
if token is None:
self.log.error("Failed to obtain required GitHub token for user '%s'" % self.options.github_user)

# options w.r.t. updating Lmod cache only make sense when Lmod is selected modules tool
if self.options.update_lmod_caches:
if self.options.modules_tool != Lmod.__name__:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be a call to the modules tool class like klass.can_create_cache or something like that

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, good suggestion

or maybe is_lmod() ;-)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a static method is_lmod in ModulesTool

self.log.error("Options related to Lmod cache are only supported if Lmod is used as a modules tool.")

self._postprocess_config()

def _postprocess_config(self):
Expand Down