From d4d5178b7944018642adfeb2c2e40455bc180569 Mon Sep 17 00:00:00 2001 From: Alexander Grund Date: Thu, 10 Jun 2021 16:19:45 +0200 Subject: [PATCH 1/3] Add docu and example for write-module-hook Also fix some typos and minor issues --- docs/Hooks.rst | 67 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 44 insertions(+), 23 deletions(-) diff --git a/docs/Hooks.rst b/docs/Hooks.rst index 40653d0d..539e49c8 100644 --- a/docs/Hooks.rst +++ b/docs/Hooks.rst @@ -59,12 +59,14 @@ Available hooks Currently (since EasyBuild v3.7.0), three types of hooks are supported: * ``start_hook`` and ``end_hook``, which are triggered *once* before starting software installations, - and *once* right after completing all installations, respectfully; -* ``parse_hook``, which is triggered when an easyconfig file is being parsed; + and *once* right after completing all installations, respectively +* ``parse_hook``, which is triggered when an easyconfig file is being parsed +* ``module_write_hook``, which is triggered right before a module file is written. + This includes the temporary module files used for extensions, sanity checks and the devel module! * "*step*" hooks that are triggered before and after every step of each installation procedure that is performed, also aptly named '``pre``'- and '``post``'-hooks -The list of currently available hooks in order of exeuction, +The list of currently available hooks in order of execution, which can also be consulted using ``eb --avail-hooks``, is: * ``start_hook`` *(only called once in an EasyBuild session)* @@ -87,11 +89,12 @@ which can also be consulted using ``eb --avail-hooks``, is: * ``pre_package_hook``, ``post_package_hook`` * ``pre_testcases_hook``, ``post_testcases_hook`` * ``end_hook`` *(only called once in an EasyBuild session)* + * ``module_write_hook`` *(can be called anytime, available since EasyBuild v4.4.1)* All functions implemented in the provided Python module for which the name ends with ``_hook`` are considered. -If any ``*_hook`` functions are encountered that do not match with any of the available hooks, an error is reported. -EasyBuild will try to provide suggestions for available hooks that closely match with the encountered unknown hook. +If any ``*_hook`` functions are encountered that do not match any of the available hooks, an error is reported. +EasyBuild will try to provide suggestions for available hooks that closely match the encountered unknown hook. For example:: @@ -119,6 +122,12 @@ Do take into account the following: * for ``parse_hook``, one argument is provided: the ``EasyConfig`` instance that corresponds to the easyconfig file being parsed (usually referred to as ``ec``) +* for ``module_write_hook``, 3 arguments are provided: + * the ``EasyBlock`` instance used to perform the installation (usually referred to as ``self``) + * the filepath of the module that will be written + * the module text as a string + The return value of this hook (if any) will then be written into the file. + * for the step hooks, one argument is provided: the ``EasyBlock`` instance used to perform the installation (usually referred to as ``self``) @@ -133,7 +142,7 @@ avoids that your hook implementations may break when updating to future EasyBuil def pre_configure_hook(self, *args, **kwargs): ... -In hooks, you have access to the full functionality provided by the EasyBuild framework, +In hooks you have access to the full functionality provided by the EasyBuild framework, so do ``import`` from ``easybuild.tools.*`` (or other ``easybuild.*`` namespaces) to leverage those functions. @@ -147,7 +156,7 @@ before further parsing of some easyconfig parameters (like ``*dependencies``) in custom data structures is done. This is important since it allows to dynamically modify easyconfig files -while they are still "raw", i.e., when the easyconfig parameter values are +while they are still "raw", i.e. when the easyconfig parameter values are still basic Python data structures like lists, dictionaries, etc. that are easy to manipulate (see also :ref:`hooks_caveats_manipulating`). @@ -161,7 +170,8 @@ Caveats ------- Due to internal details of the EasyBuild framework, you may run into some surprises when -implementing hooks. Here are some things to take into account: +implementing hooks. +Here are some things to take into account: .. _hooks_caveats_template_values: @@ -173,9 +183,10 @@ In all *step* hooks, template values in easyconfig parameters will be resolved w That is, if the ``%(version)`` template is used in for example the ``sources`` easyconfig parameter, it will be replaced with the actual value of the ``version`` easyconfig parameter whenever the ``sources`` value is used. -This can be avoided by temporarily disabling templating via ``self.cfg.enable_templating``, should the need arise. +This can be avoided by temporarily disabling templating by wrapping the code in ``with self.cfg.disable_templating:``. -There is one notable exception to this: templates in easyconfig parameters are *not* resolved in ``parse_hook``, +There is one notable exception to this: +Templates in easyconfig parameters are *not* resolved in ``parse_hook``, because templating has been disabled explicitly before ``parse_hook`` is called; this helps significantly to simplify manipulating of easyconfig parameter values (see also :ref:`hooks_caveats_manipulating`). @@ -191,7 +202,7 @@ You may run into surprises when trying to manipulate easyconfig parameters, for First of all, the original easyconfig parameters may already be processed in another data structure which does not resemble the original format in which the parameter was defined in the easyconfig file. -Moreover, this processing could be done either "in place", i.e. by replacing the original easyconfig parameter value, +Moreover, this processing could be done either "in place", i.e. by replacing the original easyconfig parameter value or in a separate variable, which effectively means that any changes to the original easyconfig parameter value are simply ignored. @@ -209,8 +220,8 @@ More specifically, the following approach will *not* work in any of the (step) h self.cfg['patches'].append('example.patch') The problem here is that the value obtained via ``self.cfg['patches']`` is not a reference -to the actual easyconfig parameter value, but a reference to a temporary copy thereof; -hence, any updates on the copy are effectively lost immediately. +to the actual easyconfig parameter value but a reference to a temporary copy thereof; +hence any updates on the copy are effectively lost immediately. To achieve the intended effect, you can either: @@ -221,12 +232,10 @@ To achieve the intended effect, you can either: def pre_fetch_hook(self): "Example of pre-fetch hook to manipulate list of patches." # temporarily disable templating, so changes to 'patches' easyconfig parameter are picked up - orig_enable_templating = self.cfg.enable_templating - self.cfg.enable_templating = False - # add patch - self.cfg['patches'].append('example.patch') - # restore templating state - self.cfg.enable_templating = orig_enable_templating + with self.cfg.disable_templating: + # add patch + self.cfg['patches'].append('example.patch') + # templating state restored * or replace the original value entirely: @@ -240,7 +249,7 @@ To achieve the intended effect, you can either: A better approach for manipulating easyconfig parameters is to use the ``parse_hook`` that was introduced in EasyBuild v3.7.0 (see :ref:`hooks_parse_hook`), where these kind of surprises will not occur (because templating is automatically disabled -before ``parse_hook`` is called, and restored immediately afterwards). +before ``parse_hook`` is called and restored immediately afterwards). See also :ref:`hooks_examples_inject_patch`. .. _hooks_caveats_archived_easyconfig: @@ -249,12 +258,12 @@ Archived easyconfig file vs hooks +++++++++++++++++++++++++++++++++ EasyBuild archives the easyconfig file that was used for a particular installation: -a copy is stored both in the ``easybuild`` subdirectory of the software installation +A copy is stored both in the ``easybuild`` subdirectory of the software installation directory and in the easyconfigs repository (see :ref:`easyconfigs_repo`). If any changes were made to the easyconfig file via hooks, these changes will *not* be -reflected in these copies. The assumption here is that the hooks will also be in place -for future (re-)installations. +reflected in these copies. +The assumption here is that the hooks will also be in place for future (re-)installations. EasyBuild does however store an additional copy of the easyconfig file which includes any modifications that were done dynamically, for example by hooks. @@ -295,3 +304,15 @@ Example hook to inject a custom patch file patch_file = 'example.patch' ec.log.info("[parse hook] Injecting additional patch file '%s'", patch_file) ec['patches'].append(patch_file) + +Example hook to replace PYTHONPATH by EBPYTHONPREFIXES in (Lua) modules +++++++++++++++++++++++++++++++++++++++++++ + +.. code:: python + + def module_write_hook(self, filepath, module_txt, *args, **kwargs): + # Note: if `self.mod_filepath == filepath` => final module file + if 'Python' in (dep['name'] for dep in self.cfg.dependencies()): + search = r'prepend_path\("PYTHONPATH", pathJoin\(root, "lib/python\d.\d/site-packages"\)\)' + replace = 'prepend_path("EBPYTHONPREFIXES", root)' + return re.sub(search, replace, module_txt) From 7ed8a40c64aca072f8adebe3ffe3f1aff451d67d Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Fri, 11 Jun 2021 21:30:36 +0200 Subject: [PATCH 2/3] minor tweaks to hooks doc page --- docs/Hooks.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/Hooks.rst b/docs/Hooks.rst index 539e49c8..b14f60db 100644 --- a/docs/Hooks.rst +++ b/docs/Hooks.rst @@ -62,7 +62,8 @@ Currently (since EasyBuild v3.7.0), three types of hooks are supported: and *once* right after completing all installations, respectively * ``parse_hook``, which is triggered when an easyconfig file is being parsed * ``module_write_hook``, which is triggered right before a module file is written. - This includes the temporary module files used for extensions, sanity checks and the devel module! + This includes the temporary module file used when installing extensions and during the sanity check, + as well as the devel module. * "*step*" hooks that are triggered before and after every step of each installation procedure that is performed, also aptly named '``pre``'- and '``post``'-hooks @@ -89,7 +90,7 @@ which can also be consulted using ``eb --avail-hooks``, is: * ``pre_package_hook``, ``post_package_hook`` * ``pre_testcases_hook``, ``post_testcases_hook`` * ``end_hook`` *(only called once in an EasyBuild session)* - * ``module_write_hook`` *(can be called anytime, available since EasyBuild v4.4.1)* + * ``module_write_hook`` *(called multiple times per installation, available since EasyBuild v4.4.1)* All functions implemented in the provided Python module for which the name ends with ``_hook`` are considered. @@ -126,7 +127,7 @@ Do take into account the following: * the ``EasyBlock`` instance used to perform the installation (usually referred to as ``self``) * the filepath of the module that will be written * the module text as a string - The return value of this hook (if any) will then be written into the file. + The return value of this hook (if any) will then be appended to the contents of the module file. * for the step hooks, one argument is provided: the ``EasyBlock`` instance used to perform the installation (usually referred to as ``self``) @@ -202,7 +203,7 @@ You may run into surprises when trying to manipulate easyconfig parameters, for First of all, the original easyconfig parameters may already be processed in another data structure which does not resemble the original format in which the parameter was defined in the easyconfig file. -Moreover, this processing could be done either "in place", i.e. by replacing the original easyconfig parameter value +Moreover, this processing could be done either "in place" by replacing the original easyconfig parameter value, or in a separate variable, which effectively means that any changes to the original easyconfig parameter value are simply ignored. @@ -306,12 +307,12 @@ Example hook to inject a custom patch file ec['patches'].append(patch_file) Example hook to replace PYTHONPATH by EBPYTHONPREFIXES in (Lua) modules -++++++++++++++++++++++++++++++++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ .. code:: python def module_write_hook(self, filepath, module_txt, *args, **kwargs): - # Note: if `self.mod_filepath == filepath` => final module file + # note: if `self.mod_filepath == filepath` => final module file if 'Python' in (dep['name'] for dep in self.cfg.dependencies()): search = r'prepend_path\("PYTHONPATH", pathJoin\(root, "lib/python\d.\d/site-packages"\)\)' replace = 'prepend_path("EBPYTHONPREFIXES", root)' From bfd9858f7d3d481e07f1c13e13a1202d80dc0616 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Fri, 11 Jun 2021 21:42:41 +0200 Subject: [PATCH 3/3] fix typo in docs on deprecated functionality --- docs/Deprecated-functionality.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Deprecated-functionality.rst b/docs/Deprecated-functionality.rst index 970085ec..ec16bf8a 100644 --- a/docs/Deprecated-functionality.rst +++ b/docs/Deprecated-functionality.rst @@ -104,7 +104,7 @@ rid of, together with supporting a better alternative. ERROR: EasyBuild encountered an exception (at easybuild/framework/easyconfig/easyconfig.py:937 in process_easyconfig): Failed to process easyconfig /home/example/gzip-1.5-goolf-1.4.10.eb: DEPRECATED (since v2.0) functionality used: Easyconfig parameter 'makeopts' is deprecated, use 'buildopts' instead; - see http://easybuild.readthedocs.org/en/latest/Deprecated-functionality.html for more informatio + see http://easybuild.readthedocs.org/en/latest/Deprecated-functionality.html for more information. * the code supporting the deprecated functionality is *removed* in EasyBuild version ``X.(Y+1)`` (i.e., the first non-bugfix-only release after version ``X.Y``), see also :ref:`removed_functionality`