diff --git a/.github/workflows/doc_build.yml b/.github/workflows/doc_build.yml index 19f45a3e..32fbe974 100644 --- a/.github/workflows/doc_build.yml +++ b/.github/workflows/doc_build.yml @@ -32,14 +32,9 @@ jobs: bash .github/workflows/check_with_codespell.sh - name: Test with sphinx run: | - pip install sphinx + pip install -r docs/requirements.txt # This test is weak, options -nWT would make it stronger but we have many warnings... READTHEDOCS=1 sphinx-build docs build - - name: Check release notes - run: | - # make sure release notes are in place for current version - ebver=$(grep '^VERSION' setup.py | sed "s/VERSION = '\([0-9.]*\)'/\1/g") - grep "^EasyBuild v${ebver}" docs/Release_notes.rst - name: Test installation run: | pip install . diff --git a/docs/Archived-easyconfigs.rst b/docs/Archived-easyconfigs.rst deleted file mode 100644 index 0e7c59db..00000000 --- a/docs/Archived-easyconfigs.rst +++ /dev/null @@ -1,78 +0,0 @@ -.. _archived_easyconfigs: - -Archived easyconfigs -==================== - -Since EasyBuild v3.0.0, easyconfig files using deprecated (i.e., old and inactive) toolchains are *archived*. - -.. contents:: - :depth: 3 - :backlinks: none - -.. _archived_easyconfigs_toolchain_deprecation: - -Toolchain deprecation ----------------------- - -Once in a blue moon, we review the list of toolchains (& versions) that are included in EasyBuild. - -Easyconfig files that use toolchains that become *deprecated* are then moved to the *easyconfigs archive*, -i.e. the ``__archive__`` subdirectory in the ``easybuild-easyconfigs`` repository -(see https://github.com/easybuilders/easybuild-easyconfigs/tree/main/easybuild/easyconfigs/__archive__). - -.. _archived_easyconfigs_deprecated_toolchains_what: - -What are deprecated toolchains? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Toolchains become deprecated if: - -* no easyconfig files using that toolchain have been contributed recently (e.g., in the last year) -* that toolchain is considered to be inactive, after consulting the EasyBuild community (via mailing list, bi-weekly conf calls) - -Deprecating a toolchain implies that all easyconfigs using that toolchain are moved to the easyconfigs archive, -and that they are no longer included in the EasyBuild regression test. -In addition, these easyconfigs are 'hidden' from plain sight, in the sense that you need to use ``--consider-archived-easyconfigs`` -to make EasyBuild consider them when it is looking for easyconfigs (e.g., with ```--search`` or ``--robot``). - -This does *not* mean that the support for using these toolchains is removed from the EasyBuild framework, -although not testing them anymore may imply that using them may no longer work at some point in time. - -For toolchains for which no active versions are available (outside of the easyconfigs archive), -it is possible that they will be reactivated, if a new toolchain version is contributed. - -.. _archived_easyconfigs_deprecated_toolchains_why: - -Why are toolchains being deprecated? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -* using old toolchains (incl. old compilers and/or libraries) is likely to become more and more difficult on modern operating systems -* these toolchains put a significant burden on the regression testing for EasyBuild releases -* easyconfigs using old toolchains are likely to be for old software versions, which may no longer be relevant anyway - -.. _archived_easyconfigs_consider: - -Using ``--consider-archived-easyconfigs`` ------------------------------------------ - -To make EasyBuild consider archived easyconfig files, you need to enable the ``--consider-archived-easyconfigs`` configuration option:: - - $ eb -S '^goolfc' - CFGS=/home/example/work/easybuild-easyconfigs/easybuild/easyconfigs - * $CFGS1/g/goolfc/goolfc-2016.08.eb - * $CFGS1/g/goolfc/goolfc-2016.10.eb - - Note: 6 matching archived easyconfig(s) found, use --consider-archived-easyconfigs to see them - -.. code:: - - $ eb -S '^goolfc' --consider-archived-easyconfigs - CFGS=/home/example/easybuild-easyconfigs/easybuild/easyconfigs - * $CFGS/g/goolfc/goolfc-2016.08.eb - * $CFGS/g/goolfc/goolfc-2016.10.eb - - Matching archived easyconfigs: - - * $CFGS/__archive__/g/goolfc/goolfc-1.3.12.eb - * $CFGS/__archive__/g/goolfc/goolfc-1.4.10.eb - * $CFGS/__archive__/g/goolfc/goolfc-2.6.10.eb diff --git a/docs/Backup_modules.rst b/docs/Backup_modules.rst deleted file mode 100644 index 3a745af2..00000000 --- a/docs/Backup_modules.rst +++ /dev/null @@ -1,98 +0,0 @@ -.. _backup_modules: - -Backing up of existing modules (``--backup-modules``) ------------------------------------------------------ - -While regenerating existing module files, you may want to preserve the existing module files -to compare and assess that the changes in the newly generated module file match expectations. - -Backing up of existing modules can be enabled with ``--backup-modules``. - -Backups are stored in the same directory as where the module file was located, -and the files are given an additional extension of the form ``.bak_``. - -* With module files in Tcl syntax, the backup module file is hidden by adding a leading dot to the filename; - this is done to avoid that it is displayed as a normal module in ``module avail``. -* With module files in Lua syntax, the backup module file is not made hidden (unless Lmod 6.x is used), - since the additional ``.bak_*`` extension prevents from picking it up as a valid module file; - only files ending in ``.lua`` are considered to be module files by Lmod 7+ . - -The location of the backed up module file will be printed, as well as a "unified diff" comparison -(very similar to what ``diff -u`` produces) between the backed up module file and the newly generated module file -(or a message mentioning that no differences were found). - - -.. _backup_modules_disable: - -Disabling automatic backup of modules -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When ``--skip`` or ``--module-only`` is used, backing up of existing modules is enabled automatically. - -This can be disabled with ``--disable-backup-modules``. - - -.. _backup_modules_example: - -Example -~~~~~~~ - -Suppose existing modules in both Tcl & Lua syntax are present (``bzip2/1.0.6``). - -Both these module files are missing an update statement for $PATH -because the ``/bin`` subdirectory was missing in the installation, -for the purpose of this example:: - - $ ls -la $EASYBUILD_PREFIX/modules/all/bzip2 - total 16 - drwxr-xr-x 2 example example 136 Aug 24 10:20 . - drwxr-xr-x 3 example example 102 Aug 24 10:18 .. - -rw-r--r-- 1 example example 1256 Aug 24 10:19 1.0.6 - -rw-r--r-- 1 example example 1303 Aug 24 10:18 1.0.6.lua - - -Using ``--force`` and ``--backup-modules``, we can reinstall the ``bzip2/1.0.6`` modules -and get a clear view on what has changed. - -To reinstall the ``bzip2/1.0.6`` module in Lua syntax while retaining a backup of the existing module:: - - $ eb bzip2-1.0.6.eb --module-syntax=Lua --force --backup-modules - ... - == creating build dir, resetting environment... - == backup of existing module file stored at /example/modules/all/bzip2/1.0.6.lua.bak_20170824102603 - ... - == creating module... - == comparing module file with backup /example/modules/all/bzip2/1.0.6.lua.bak_20170824102603; diff is: - --- /example/modules/all/bzip2/1.0.6.lua.bak_20170824102603 - +++ /example/modules/all/bzip2/1.0.6.lua - @@ -25,9 +25,10 @@ - prepend_path("LD_LIBRARY_PATH", pathJoin(root, "lib")) - prepend_path("LIBRARY_PATH", pathJoin(root, "lib")) - prepend_path("MANPATH", pathJoin(root, "man")) - +prepend_path("PATH", pathJoin(root, "bin")) - setenv("EBROOTBZIP2", root) - setenv("EBVERSIONBZIP2", "1.0.6") - setenv("EBDEVELBZIP2", pathJoin(root, "easybuild/bzip2-1.0.6-easybuild-devel")) - - ... - -Equivalently, we can reinstall the module in Tcl syntax using:: - - $ eb bzip2-1.0.6.eb --module-syntax=Tcl --force --backup-modules - -Afterwards, both the newly generated modules and the backups are in place:: - - $ ls -la $EASYBUILD_PREFIX/modules/all/bzip2 - total 32 - drwxr-xr-x 2 example example 204 Aug 24 10:26 . - drwxr-xr-x 3 example example 102 Jul 11 10:18 .. - -rw-r--r-- 1 example example 1227 Aug 24 10:24 .1.0.6.bak_20170824102412 - -rw-r--r-- 1 example example 1256 Jul 11 01:24 1.0.6 - -rw-r--r-- 1 example example 1303 Jul 11 01:26 1.0.6.lua - -rw-r--r-- 1 example example 1259 Aug 24 10:26 1.0.6.lua.bak_20170824102603 - -Cleaning up the backup module files can be done with the following command (for example):: - - $ find $EASYBUILD_PREFIX/modules/all/bzip2 -name '*.bak*' | xargs rm -v - /example/modules/all/bzip2/.1.0.6.bak_20170824102412 - /example/modules/all/bzip2/1.0.6.lua.bak_20170824102603 diff --git a/docs/Changelog.rst b/docs/Changelog.rst deleted file mode 100644 index 4d34fe0a..00000000 --- a/docs/Changelog.rst +++ /dev/null @@ -1,253 +0,0 @@ -.. _changelog: - -Changelog for EasyBuild documentation -------------------------------------- - -(for EasyBuild release notes, see :ref:`release_notes`) - -* **release 20230109.01** (`Jan 9th 2023`): update release notes for EasyBuild v4.7.0 (see :ref:`release_notes_eb470`) -* **release 20221021.01** (`Oct 21st 2022`): update release notes for EasyBuild v4.6.2 (see :ref:`release_notes_eb462`) -* **release 20220912.01** (`Sep 12th 2022`): update release notes for EasyBuild v4.6.1 (see :ref:`release_notes_eb461`) -* **release 20220708.01** (`Jul 8th 2022`): update release notes for EasyBuild v4.6.0 (see :ref:`release_notes_eb460`) -* **release 20220608.01** (`Jun 8th 2022`): update release notes for EasyBuild v4.5.5 (see :ref:`release_notes_eb455`) -* **release 20220331.01** (`Mar 31st 2022`): update release notes for EasyBuild v4.5.4 (see :ref:`release_notes_eb454`) -* **release 20220211.01** (`Feb 11th 2022`): update release notes for EasyBuild v4.5.3 (see :ref:`release_notes_eb453`) -* **release 20220124.01** (`Jan 24th 2022`): update release notes for EasyBuild v4.5.2 (see :ref:`release_notes_eb452`) -* **release 20211213.01** (`Dec 13th 2021`): update release notes for EasyBuild v4.5.1 (see :ref:`release_notes_eb451`) -* **release 20211029.01** (`Oct 29th 2021`): update release notes for EasyBuild v4.5.0 (see :ref:`release_notes_eb450`) -* **release 20210907.01** (`Sep 7th 2021`): update release notes for EasyBuild v4.4.2 (see :ref:`release_notes_eb442`) -* **release 20210706.01** (`Jul 6th 2021`): update release notes for EasyBuild v4.4.1 (see :ref:`release_notes_eb441`) -* **release 20210602.01** (`Jun 2nd 2021`): update release notes for EasyBuild v4.4.0 (see :ref:`release_notes_eb440`) -* **release 20210409.01** (`Apr 9th 2021`): update release notes for EasyBuild v4.3.4 (see :ref:`release_notes_eb434`) -* **release 20210223.01** (`Feb 23rd 2021`): update release notes for EasyBuild v4.3.3 (see :ref:`release_notes_eb433`) -* **release 20201210.01** (`Dec 10th 2020`): update release notes for EasyBuild v4.3.2 (see :ref:`release_notes_eb432`) -* **release 20201029.01** (`Oct 29th 2020`): update release notes for EasyBuild v4.3.1 (see :ref:`release_notes_eb431`) -* **release 20200913.01** (`Sep 13th 2020`): update release notes for EasyBuild v4.3.0 (see :ref:`release_notes_eb430`) -* **release 20200708.01** (`July 8th 2020`): update release notes for EasyBuild v4.2.2 (see :ref:`release_notes_eb422`) -* **release 20200520.01** (`May 20th 2020`): update release notes for EasyBuild v4.2.1 (see :ref:`release_notes_eb421`) -* **release 20200414.01** (`Apr 14th 2020`): - - * document new EasyBuild locking mechanism (see :ref:`locks`) - * document support for creating index files (see :ref:`easyconfigs_index`) - * update release notes for EasyBuild v4.2.0 (see :ref:`release_notes_eb420`) -* **release 20200316.01** (`Mar 16th 2020`): update release notes for EasyBuild v4.1.2 (see :ref:`release_notes_eb412`) -* **release 20200116.01** (`Jan 16th 2020`): update release notes for EasyBuild v4.1.1 (see :ref:`release_notes_eb411`) -* **release 20191204.01** (`Dec 4th 2019`): update release notes for EasyBuild v4.1.0 (see :ref:`release_notes_eb410`) -* **release 20191015.01** (`Oct 15th 2019`): update release notes for EasyBuild v4.0.1 (see :ref:`release_notes_eb401`) -* **release 20190920.01** (`Sep 20th 2019`): update release notes for EasyBuild v4.0.0 (see :ref:`release_notes_eb400`); see also :ref:`eb4_changes_overview` -* **release 20190823.01** (`Aug 23rd 2019`): update release notes for EasyBuild v3.9.4 (see :ref:`release_notes_eb394`) -* **release 20190708.01** (`Jul 8th 2019`): update release notes for EasyBuild v3.9.3 (see :ref:`release_notes_eb393`) -* **release 20190609.01** (`Jun 9th 2019`): update release notes for EasyBuild v3.9.2 (see :ref:`release_notes_eb392`) -* **release 20190908.01** (`Jun 8th 2019`): update documentation on support for generating (Singularity) container recipes/images (see :ref:`containers`) -* **release 20190520.01** (`May 20th 2019`): update release notes for EasyBuild v3.9.1 (see :ref:`release_notes_eb391`) -* **release 20190412.01** (`Apr 12th 2019`): update release notes for EasyBuild v3.9.0 (see :ref:`release_notes_eb390`) -* **release 20190129.01** (`Jan 29th 2019`): update release notes for EasyBuild v3.8.1 (see :ref:`release_notes_eb381`) -* **release 20190124.01** (`Jan 24th 2019`): add definitions for ``2019a`` common toolchains (see :ref:`common_toolchains`) -* **release 20181218.01** (`Dec 18th 2018`): update release notes for EasyBuild v3.8.0 (see :ref:`release_notes_eb380`) -* **release 20181114.01** (`Nov 14th 2018`): add documentation on deprecated easyconfigs & toolchains (:ref:`deprecated_easyconfigs`) -* **release 20181104.01** (`Nov 4th 2018`): update ``--job`` documentation to also cover ``Slurm`` job backend (see :ref:`submitting_jobs`) -* **release 20181018.01** (`Oct 18th 2018`): update release notes for EasyBuild v3.7.1 (see :ref:`release_notes_eb371`) -* **release 20180925.01** (`Sep 25th 2018`): update release notes for EasyBuild v3.7.0 (see :ref:`release_notes_eb370`) -* **release 20180921.01** (`Sep 21st 2018`): document support for wrapping dependencies (see :ref:`wrapping_dependencies`) -* **release 20180920.01** (`Sep 20th 2018`): document support for downloading sources directly from a Git repository (see :ref:`common_easyconfig_param_sources_git_config`) -* **release 20180711.01** (`Jul 11th 2018`): update release notes for EasyBuild v3.6.2 (see :ref:`release_notes_eb362`) -* **release 20180710.01** (`Jul 10th 2018`): add definitions for ``2018b`` common toolchains (see :ref:`common_toolchains`) -* **release 20180528.01** (`May 28th 2018`): update release notes for EasyBuild v3.6.1 (see :ref:`release_notes_eb361`) -* **release 20180427.01** (`Apr 27th 2018`): update release notes for EasyBuild v3.6.0 (see :ref:`release_notes_eb360`) -* **release 20180425.01** (`Apr 25th 2018`): add documentation on support for generating container recipes & images (see :ref:`containers`) -* **release 20180307.01** (`Mar 7th 2018`): update release notes for EasyBuild v3.5.3 (see :ref:`release_notes_eb353`) -* **release 20180302.01** (`Mar 2nd 2018`): update release notes for EasyBuild v3.5.2 (see :ref:`release_notes_eb352`) -* **release 20180116.01** (`Jan 16th 2018`): update release notes for EasyBuild v3.5.1 (see :ref:`release_notes_eb351`) -* **release 20180112.01** (`Jan 12th 2018`): add definitions for ``2018a`` common toolchains (see :ref:`common_toolchains`) -* **release 20171215.01** (`Dec 15th 2017`): update release notes for EasyBuild v3.5.0 (see :ref:`release_notes_eb350`) -* **release 20171208.01** (`Dec 8th 2017`): document support for user-defined hooks (see :ref:`hooks`) -* **release 20171017.01** (`Oct 17th 2017`): update release notes for EasyBuild v3.4.1 (see :ref:`release_notes_eb341`) -* **release 20170910.01** (`Sept 10th 2017`): update release notes for EasyBuild v3.4.0 (see :ref:`release_notes_eb340`) -* **release 20170906.01** (`Sept 6th 2017`): document ``--trace`` (see :ref:`trace`) -* **release 20170824.02** (`Aug 24th 2017`): document ``--inject-checksums`` (see :ref:`inject_checksums`) -* **release 20170824.01** (`Aug 24th 2017`): document ``--backup-modules`` (see :ref:`backup_modules`) -* **release 20170712.01** (`July 12th 2017`): update release notes for EasyBuild v3.3.1 (see :ref:`release_notes_eb331`) -* **release 20170708.01** (`July 8th 2017`): add documentation on :ref:`github_merge_pr` -* **release 20170705.01** (`July 5th 2017`): clarify :ref:`contributing_review_process_pr_requirements`, add page listing :ref:`maintainers` -* **release 20170626.01** (`June 26th 2017`): update release notes for EasyBuild v3.3.0 (see :ref:`release_notes_eb330`) -* **release 20170623.01** (`June 23rd 2017`): document use of ``checksums`` & alternative formats for ``sources`` (see :ref:`common_easyconfig_param_sources`) -* **release 20170622.01** (`June 22nd 2017`): document support for detecting loaded modules (see :ref:`detect_loaded_modules`) -* **release 20170522.01** (`May 22nd 2017`): document deprecated behaviour in EasyBuild v3.2.0 (see :ref:`overview_deprecated`) -* **release 20170512.01** (`May 12th 2017`): update release notes for EasyBuild v3.2.1 (see :ref:`release_notes_eb321`) -* **release 20170505.01** (`May 5th 2017`): update release notes for EasyBuild v3.2.0 (see :ref:`release_notes_eb320`) -* **release 20170320.01** (`Mar 20th 2017`): update release notes for EasyBuild v3.1.2 (see :ref:`release_notes_eb312`) -* **release 20170307.01** (`Mar 7th 2017`): update release notes for EasyBuild v3.1.1 (see :ref:`release_notes_eb311`) -* **release 20170221.01** (`Feb 21st 2017`): add documentation on :ref:`contributing` -* **release 20170209.01** (`Feb 9th 2017`): add documentation on implementing easyblocks (see :ref:`implementing_easyblocks`) -* **release 20170203.01** (`Feb 3rd 2017`): update release notes for EasyBuild v3.1.0 (see :ref:`release_notes_eb310`) -* **release 20170129.01** (`Jan 29th 2017`): update ``--optarch`` documentation (see :ref:`controlling_compiler_optimization_flags_optarch_per_compiler`) -* **release 20170109.01** (`Jan 9th 2017`): add documentation on :ref:`common_toolchains` -* **release 20161222.01** (`Dec 22nd 2016`): update documentation and release notes for EasyBuild v3.0.2 (see :ref:`release_notes_eb302`) -* **release 20161218.01** (`Dec 18th 2016`): document need to download ``vsc-*`` source tarballs from PyPI (see :ref:`bootstrap_offline`) -* **release 20161202.01** (`Dec 2nd 2016`): add documentation on Cray support (see :ref:`cray_support`) -* **release 20161130.01** (`Nov 16th 2016`): update release notes for EasyBuild v3.0.1 (see :ref:`release_notes_eb301`) -* **release 20161117.01** (`Nov 17th 2016`): update mentions of default configuration according to updated default in EasyBuild v3.0.0 -* **release 20161116.01** (`Nov 16th 2016`): update documentation and release notes for EasyBuild v3.0.0 (see :ref:`release_notes_eb300`) - - * :ref:`archived_easyconfigs` - * :ref:`rpath_support` - -* **release 20161028.01** (`Oct 28th 2016`): recommend using ``--rebuild`` rather than ``--force`` (see :ref:`rebuild_option`) -* **release 20161023.01** (`Oct 24th 2016`): add section on iterating over configure/build/install options (see :ref:`configure_build_install_command_options_iterate`) -* **release 20161014.01** (`Oct 14th 2016`): update documentation on deprecated functionality (see :ref:`deprecated`) -* **release 20161010.01** (`Oct 10th 2016`): add page for EasyBuild demos (see :ref:`demos`) -* **release 20160923.02** (`Sept 23rd 2016`): update release notes for EasyBuild v2.9.0 (see :ref:`release_notes_eb290`) -* **release 20160923.01** (`Sept 23rd 2016`): add generated list of supported software (see :ref:`list_software`) -* **release 20160713.01** (`July 13th 2016`): update release notes for EasyBuild v2.8.2 (see :ref:`release_notes_eb282`) -* **release 20160613.01** (`June 13th 2016`): clarify required dependencies (setuptools, vsc-install) (see :ref:`required_python_packages`) -* **release 20160607.01** (`June 7th 2016`): update/complete documentation on GitHub integration (see :ref:`integration_with_github`) -* **release 20160530.01** (`May 30th 2016`): update release notes for EasyBuild v2.8.1 (see :ref:`release_notes_eb281`) -* **release 20160518.01** (`May 18th 2016`): update release notes for EasyBuild v2.8.0 (see :ref:`release_notes_eb280`) -* **release 20160429.01** (`April 29th 2016`): add section on updating EasyBuild, see :ref:`updating` -* **release 20160320.01** (`March 20th 2016`): update release notes for EasyBuild v2.7.0 (see :ref:`release_notes_eb270`) -* **release 20160228.01** (`February 28th 2016`): - - * update documentation on external modules metadata (see :ref:`using_external_modules_metadata`) - -* **release 20160214.01** (`February 14th 2016`): - - * add section on ``--show-config`` (see :ref:`configuration_show_config`) - -* **release 20160126.02** (`January 26th 2016`): packaging support is stable since EasyBuild v2.5.0 (see :ref:`packaging_support`) -* **release 20160126.01** (`January 26th 2016`): update release notes for EasyBuild v2.6.0 (see :ref:`release_notes_eb260`) -* **release 20151217.01** (`December 17th 2015`): update release notes for EasyBuild v2.5.0 (see :ref:`release_notes_eb250`) -* **release 20151209.01** (`December 9th 2015`): - - * add documentation on controlling compiler optimizations flags, see :ref:`controlling_compiler_optimization_flags` - -* **release 20151110.01** (`November 10th 2015`): update release notes for EasyBuild v2.4.0 (see :ref:`release_notes_eb240`) -* **release 20151108.01** (`November 8th 2015`): - - * document (experimental) support for using minimal toolchains (see :ref:`minimal_toolchains`) - -* **release 20151028.01** (`October 28th 2015`): document extended dry run mechanism (see :ref:`extended_dry_run`) -* **release 20151021.01** (`October 21st 2015`): - - * include initial documentation on experimental support for easyconfig files in YAML syntax (``.yeb``), - see :ref:`easyconfig_yeb_format` - -* **release 20150902.01** (`September 2nd 2015`): update release notes for EasyBuild v2.3.0 (see :ref:`release_notes_eb230`) -* **release 20150715.01** (`July 15th 2015`): update release notes for EasyBuild v2.2.0 (see :ref:`release_notes_eb220`) -* **release 20150714.01** (`July 14th 2015`): add documentation on :ref:`packaging_support` -* **release 20150709.01** (`July 9th 2015`): add documentation on :ref:`submitting_jobs` -* **release 20150708.01** (`July 8th 2015`): - - * add documentation on ``--include-*`` options (see :ref:`including_additional_python_modules`) - -* **release 20150703.01** (`July 3rd 2015`): - - * fix outdated documentation on ``easyblock`` parameter (see :ref:`writing_easyconfigs_easyblock_spec`) - -* **release 20150624.01** (`June 24th 2015`): mention ``MigrateFromEBToHMNS`` module naming scheme in section on - ``--module-only`` (see :ref:`module_only_additional`) -* **release 20150610.01** (`June 10th 2015`): update :ref:`installing_lmod` for Lmod v6.0 - -* **release 20150518.01** (`May 18th 2015`): - - * update section on ``--search``: better examples + highlight ability to search via regular expression (see :ref:`searching_for_easyconfigs`) - * update release notes for EasyBuild v2.1.1 (see :ref:`release_notes_eb211`) - -* **release 20150506.01** (`May 6th 2015`): updated documentation for EasyBuild v2.1.1 - - * add note on ``$LMOD_CMD`` fallback to find full path to ``lmod`` binary (see :ref:`required_modules_tool`) - -* **release 20150430.01** (`Apr 30th 2015`): updated documentation for EasyBuild v2.1.0 - - * also cover extensions in page on concepts and terminology (see :ref:`extensions`) - * add documentation on :ref:`partial_installations`, covering ``--stop``, ``--skip`` and ``--module-only`` - * add documentation on :ref:`manipulating_dependencies`, covering ``--filter-deps`` and ``--hide-deps`` - * document ``-module-syntax`` configuration option (see :ref:`module_syntax`) - * add note on detection of unknown ``$EASYBUILD``-prefixed environment variables (see :ref:`configuration_env_vars`) - * mention support for prepending/appending to ``--robot-paths`` (see :ref:`robot_search_path_prepend_append`) - * update release notes for EasyBuild v2.1.0 (see :ref:`release_notes`) - -* **release 20150425.01** (`Apr 25th 2015`): - - * add documentation on :ref:`using_external_modules` - -* **release 20150407.01** (`Apr 7th 2015`): - - * add link to :ref:`unit_tests` page in dedicated section at :ref:`installation` page - (see :ref:`install_running_unit_tests`) - * clarify relation between ``--installpath``, ``--prefix``, ``-subdir-*`` and ``--installpath-*`` - configuration options (see :ref:`installpath`) - * mention ``--show-default-configfiles`` command line option in relevant section - (see :ref:`default_configuration_files`) - -* **release 20150327.01** (`Mar 27th 2015`): - - * documented deprecated functionality w.r.t. error reporting (see :ref:`depr_error_reporting`) -* **release 20150316.01** (`Mar 16th 2015`): - - * include list of EasyBuild repositories cloned by ``install-EasyBuild-develop.sh`` script - (see :ref:`install_latest_develop_using_script`) -* **release 20150312.01** (`Mar 12th 2015`): - - * enhance documentation w.r.t. template values in configuration files (see :ref:`configuration_file_templates_constants`) - * improve documentation on ``--robot`` and ``--robot-paths`` (see :ref:`controlling_robot_search_path`) -* **release 20150310.01** (`Mar 10th 2015`): - - * document peculiarties w.r.t. dependencies in combination with a ``dummy`` toolchain (see :ref:`dependency_specs`) - * document ``clean_gists.py`` script (see :ref:`clean_gists_script`) - * mention taking into account of proxy settings for downloading sources (see :ref:`common_easyconfig_param_sources`) -* **release 20150306.03** (`Mar 6th 2015`): add release notes for EasyBuild v2.0.0 (see :ref:`release_notes`) -* **release 20150306.02** (`Mar 6th 2015`): - - * add documentation on GitHub integration features (see :ref:`integration_with_github`), mainly ``--from-pr`` (see :ref:`from_pr`) - * document locations where (specified) easyconfig files are being searched for (see :ref:`specifying_easyconfigs`) -* **release 20150306.01** (`Mar 6th 2015`): - - * add documentation on removed functionality (see :ref:`removed_functionality`) - * clean up documentation on deprecated functionality (see :ref:`deprecated`) - * add documentation on provided scripts, in particular ``fix-broken-easyconfigs.py`` (see :ref:`useful_scripts`) -* **release 20150302.01** (`Mar 2nd 2015`): update/cleanup documentation on :ref:`alt_inst_methods` -* **release 20150227.02** (`Feb 27th 2015`): add documentation on the EasyBuild unit test suites, see :ref:`unit_tests` -* **release 20150227.01** (`Feb 27th 2015`): enhance documentation w.r.t. to (optional dependencies), see :ref:`installation` -* **release 20150220.01** (`Feb 20th 2015`): - - * document new advanced bootstrapping options: skipping stage 0 and providing source tarballs (see :ref:`installation`) -* **release 20150219.01** (`Feb 19th 2015`): first updates for EasyBuild v2.0.0 - - * extend section on (default) EasyBuild configuration files to also cover ``$XDG_CONFIG_DIRS`` (see :ref:`configuration_file:`) -* **release 20150205.01** (`Feb 5th 2015`): include information on deprecated functionality in (generic) easyblocks (see :ref:`deprecated`) -* **release 20150126.01** (`Jan 26th 2015`): - - * fix ``pip`` installation prefix option (:ref:`alt_inst_methods`) - * clarify need to have modules tool binary available in ``$PATH`` (:ref:`installation`) -* **release 20150112.01** (`Jan 12th 2015`): mention need to escape ``%`` when setting log file format via config file (see :ref:`logfile_format`) -* **release 20150107.01** (`Jan 7th 2015`): document behaviour of `dummy` toolchain (:ref:`dummy_toolchain`) -* **release 20141219.01** (`Dec 19th 2014`): add release notes for EasyBuild v1.16.1 (see :ref:`release_notes`) -* **release 20141218.01** (`Dec 18th 2014`): add release notes for EasyBuild v1.16.0 (see :ref:`release_notes`) -* **release 20141217.01** (`Dec 17th 2014`): document deprecated functionality in EasyBuild v1.x (:ref:`deprecated`) -* **release 20141204.02** (`Dec 4th 2014`): add EasyBuild release notes (see :ref:`release_notes`) -* **release 20141204.01** (`Dec 4th 2014`): updates for EasyBuild v1.16.0 - - * document details w.r.t. (controlling of) robot search path, incl. ``--robot-paths`` (:ref:`using_the_easybuild_command_line`) - * document use of templates and constants in EasyBuild configuration files (:ref:`configuring_easybuild`) - * bump EasyBuild version to 1.16.0 - * changed release number scheme for documentation (based on datestamp) -* **release 1.0.3** (`Dec 3rd 2014`): add page on :ref:`code_style` -* **release 1.0.2** (`Nov 6th 2014`): typo and grammar fixes, update Lmod installation instructions for Lmod v5.8 -* **release 1.0.1** (`Nov 4th 2014`): fix issues with Changelog -* **release 1.0.0** `(Nov 4th 2014)`: initial release of revamped EasyBuild documentation - @ http://easybuild.readthedocs.org, covering basic topics: - - * introductory topics: - - * :ref:`what_is_easybuild` - * :ref:`concepts_and_terminology` - * :ref:`typical_workflow` - * getting started: - - * :ref:`installation` - * :ref:`configuring_easybuild` - * basic usage topics: - - * :ref:`using_the_easybuild_command_line` - * :ref:`writing_easyconfig_files` - * :ref:`understanding_easyBuild_logs` diff --git a/docs/Code_style.rst b/docs/Code_style.rst deleted file mode 100644 index b29decaf..00000000 --- a/docs/Code_style.rst +++ /dev/null @@ -1,38 +0,0 @@ - -.. _code_style: - -Code style -========== - -The code style we follow in the EasyBuild code repository is mainly dictated by the Python standard `PEP8`_. - -Highlighted PEP8 code style rules: - -* use **4 spaces** for indentation, **do not use tabs** - for example, use ``:set tabstop 4`` and ``:set expandtab`` in Vim -* indent items in a list at an extra 4 spaces - nested lists can be indented at the same indentation as the first item in the list if it is on the first line, closing brackets must match visual indentation -* use `optional underscores`, not camelCase, for variable, function and method names (i.e. ``poll.get_unique_voters()``, - **not** ``poll.getUniqueVoters``) -* use ``InitialCaps`` for class names -* in docstrings, don't use "action words" - -The only (major) exception to PEP8 is our preference for longer line lengths: line lengths **must be limited to 120 characters**, and should by preference be `shorter than 100 characters` (as opposed to the 80-character limit in PEP8). - -.. _PEP8: http://www.python.org/dev/peps/pep-0008 - - -Notes -~~~~~ - -Code style in easyconfig files can be **automatically checked** using ``--check-contrib``, -for example: ``eb --check-contrib sympy-1.3-intel-2018a-Python-2.7.14.eb`` -(see :ref:`contributing_review_process_code_style` for more details). - -Style guides that go a step beyond PEP8: - * http://www.gramps-project.org/wiki/index.php?title=Programming_guidelines - * http://code.google.com/p/volatility/wiki/StyleGuide - -Automatic rewriting of Python code: http://pypi.python.org/pypi/PythonTidy/1.22 - -``pep8`` might be a useful tool to check PEP8 compliance: https://github.com/jcrocholl/pep8 diff --git a/docs/Common-toolchains.rst b/docs/Common-toolchains.rst deleted file mode 100644 index d2e988b6..00000000 --- a/docs/Common-toolchains.rst +++ /dev/null @@ -1,279 +0,0 @@ -.. _common_toolchains: - -Common toolchains -================= - -This page documents the concept of *common toolchains* in the EasyBuild community; -for a more general definition of what (compiler) toolchains are, see :ref:`toolchains`. - -.. contents:: - :depth: 3 - :backlinks: none - - -.. _common_toolchains_what: - -Definition and motivation -------------------------- - -Picking a :ref:`compiler toolchain ` to use is one of the first things you (need to) do when starting to use EasyBuild. -This can be a daunting task, since a whole bunch of toolchains and different toolchain versions -are readily available in EasyBuild. It may be difficult to determine which toolchain would be most rewarding to use, -in terms of stability, performance of the resulting binaries and readily available easyconfig files. - -In an attempt to focus the effort of the EasyBuild community, -the concept of so-called *common toolchains* was introduced. - -The idea is to compose and maintain a limited set of specific compiler toolchains, -and try and convince many HPC sites to employ these toolchains. -This helps in assuring stability of these toolchains w.r.t. which software can be built (correctly) with them, -since they get significantly more testing. In addition, the expectation/hope is that more easyconfigs are -contributed back to the central easyconfigs repository (https://github.com/easybuilders/easybuild-easyconfigs), -resulting in a wide range of readily available easyconfig files using the common toolchains. - -The intention is to revise/update the definitions of the common toolchains regularly -(see :ref:`common_toolchains_update_cycle`), which again can be a joint effort -that benefits many HPC sites. - -Currently, two different common toolchains are being maintained: ``foss`` and ``intel``; -see below for more details, and also :ref:`common_toolchains_overview`. - - -.. _common_toolchains_foss: - -``foss`` toolchain -~~~~~~~~~~~~~~~~~~ - -The ``foss`` common compiler toolchain consists entirely of open source software (hence the name, -derived from the common term 'FOSS', which is short for "Free and Open Source Software"). - -This toolchain consists of: - -* binutils (https://www.gnu.org/software/binutils/) - -* the GNU Compiler Collection (GCC, https://gcc.gnu.org/), - i.e. ``gcc`` (C), ``g++`` (C++) and ``gfortran`` (Fortran) - -* the Open MPI library (https://www.open-mpi.org/) - -* the OpenBLAS (http://www.openblas.net/) + LAPACK (http://netlib.org/lapack) libraries (for ``foss`` < 2021a) - -* the FlexiBLAS library (https://www.mpi-magdeburg.mpg.de/projects/flexiblas), with OpenBLAS + LAPACK as backend (for ``foss`` >= 2021a) - -* the ScaLAPACK (http://netlib.org/scalapack) library is also included - -* the FFTW library (http://fftw.org/) - -.. note:: The toolchain name was deliberately chosen to be generic, to allow for swapping - any of the toolchain components with a better (open source) alternative in the future, - should the need or opportunity arise. - - -.. _common_toolchains_intel: - -``intel`` toolchain -~~~~~~~~~~~~~~~~~~~ - -The ``intel`` common compiler toolchain consists of the Intel compilers and libraries, i.e.: - -* the Intel C/C++/Fortran compilers (https://software.intel.com/en-us/intel-compilers), - i.e. ``icc``, ``icpc`` and ``ifort``, - - * binutils and GCC, which serve as a base for the Intel compilers, are also included - -* the Intel MPI library (https://software.intel.com/en-us/intel-mpi-library) - -* the Intel Math Kernel Library (MKL, https://software.intel.com/en-us/intel-mkl) for BLAS/LAPACK/FFT functionality - -.. note:: This compiler toolchain includes licensed software; - valid licenses must be available to install and use it. - - -.. _common_toolchains_versioning_scheme: - -Versioning scheme for common toolchains ---------------------------------------- - -The common toolchains follow a specific versioning scheme, which takes the -:ref:`6-month update cycle ` into account. - -Each revision of the common toolchains is versioned as the *year* in which it -was defined, plus an additional '``a``' or '``b``' to indicate whether the toolchain -was defined at the start of the year ('``a``') or halfway through the year ('``b``'); -in short, the common toolchains are versioned as ``{a,b}``. - -For example, ``foss/2016b`` is a revision of the ``foss`` that was composed mid-2016. - -A full historic overview of the ``foss`` and ``intel`` common toolchains is -available in :ref:`common_toolchains_overview`. - -.. note:: Next to the versions that follow the ``{a,b}`` versioning scheme, - additional versions of the ``foss`` and ``intel`` versions are available - as well. - - These versions are **not** considered to be part of the series of common - toolchains (even though they consists of the same toolchain components). - These versions may be site-specific, or compositions that were put in - place to evaluate a potential future common toolchain. - - Typically, they are versioned as ``.``, indicating when - the most recent component included was released, or when that particular - toolchain composition was defined. - - -.. _common_toolchains_update_cycle: - -Update cycle for common toolchains ----------------------------------- - -The intention is to revise and update the common toolchains every 6 months: -once in late December/early January (version ``a``), -and once in late June/early July (version ``b``). - -This is meant be to be a community effort, in the sense that a proposal -for an updated composition is shared and discussed before it is set in stone. - -Recent versions of each of the toolchain components are considered, taking -into account stability, performance improvements, added features, -known bugs/issues and experiences with those versions. - -Moreover, the proposed toolchain compositions are tested extensively, -typically by rebuilding all available easyconfigs that are using the -most recent revision of the common toolchains at that time. - -.. _common_toolchains_overview: - -Overview of common toolchains ------------------------------ - -.. _common_toolchains_overview_foss: - -Component versions in ``foss`` toolchain -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``foss`` | *date* |*binutils* | *GCC* | *Open MPI* | *FlexiBLAS* | *OpenBLAS* | *LAPACK* | *ScaLAPACK* | *FFTW* | -+================+=============+============+===========+============+=============+============+=======================+==============+=============+ -| ``2019a`` | Jan '19 | 2.31.1 | 8.2.0 | 3.1.3 | *(none)* | 0.3.5 | (incl. with OpenBLAS) | 2.0.2 | 3.3.8 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2019b`` | Sept '19 | 2.32 | 8.3.0 | 3.1.4 | *(none)* | 0.3.7 | (incl. with OpenBLAS) | 2.0.2 | 3.3.8 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2020a`` | May '20 | 2.34 | 9.3.0 | 4.0.3 | *(none)* | 0.3.9 | (incl. with OpenBLAS) | 2.1.0 | 3.3.8 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2020b`` | Nov '20 | 2.35 | 10.2.0 | 4.0.5 | *(none)* | 0.3.12 | (incl. with OpenBLAS) | 2.1.0 | 3.3.8 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2021a`` | May '21 | 2.36.1 | 10.3.0 | 4.1.1 | 3.0.4 | 0.3.15 | (incl. with OpenBLAS) | 2.1.0 | 3.3.9 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2021b`` | Oct '21 | 2.37 | 11.2.0 | 4.1.1 | 3.0.4 | 0.3.18 | (incl. with OpenBLAS) | 2.1.0 | 3.3.10 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2022a`` | Jun '22 | 2.38 | 11.3.0 | 4.1.4 | 3.2.0 | 0.3.20 | (incl. with OpenBLAS) | 2.2.0 | 3.3.10 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2022b`` | Dec '22 | 2.39 | 12.2.0 | 4.1.4 | 3.2.1 | 0.3.21 | (incl. with OpenBLAS) | 2.2.0 | 3.3.10 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ - -*(components marked with* * *were patched)* - -.. _common_toolchains_overview_intel: - -Component versions in ``intel`` toolchain -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``intel`` | *date* | *binutils* | *GCC* | *Intel compilers* | *Intel MPI* | *Intel MKL* | -+================+=============+=============+===========+====================+=============+==============+ -| ``2019a`` | Jan '19 | 2.31.1 | 8.2.0 | 2019.1.144 | 2018.4.274 | 2019.1.144 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2019b`` | Sept '19 | 2.32 | 8.3.0 | 2019.5.281 | 2018.5.288 | 2019.5.281 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2020a`` | May'20 | 2.34 | 9.3.0 | 2020.1.217 | 2019.7.217 | 2020.1.217 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2020b`` | Nov'20 | 2.35 | 10.2.0 | 2020.4.304 | 2019.9.304 | 2020.4.304 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2021a`` | May'21 | 2.36.1 | 10.3.0 | 2021.2.0 | 2021.2.0 | 2021.2.0 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2021b`` | Oct'21 | 2.37 | 11.2.0 | 2021.4.0 | 2021.4.0 | 2021.4.0 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2022a`` | Jun'22 | 2.38 | 11.3.0 | 2022.1.0 | 2021.6.0 | 2022.1.0 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2022b`` | Dec'22 | 2.39 | 12.2.0 | 2022.2.1 | 2021.7.1 | 2022.2.1 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ - -.. _common_toolchains_overview_deprecated: - -Overview of common toolchains (deprecated versions) ---------------------------------------------------- - -.. _common_toolchains_overview_foss_deprecated: - -Component versions in ``foss`` toolchain (deprecated versions) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``foss`` | *date* |*binutils* | *GCC* | *Open MPI* | *FlexiBLAS* | *OpenBLAS* | *LAPACK* | *ScaLAPACK* | *FFTW* | -+================+=============+============+===========+============+=============+============+=======================+==============+=============+ -| ``2014b`` | Jul '14 | '*(none)* | 4.8.3 | 1.8.1 | *(none)* | 0.2.9 | 3.5.0 | 2.0.2 | 3.3.4 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2015a`` | Jan '15 | '*(none)* | 4.9.2 | 1.8.4 | *(none)* | 0.2.13 | 3.5.0 | 2.0.2 | 3.3.4 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2015b`` | Jul '15 | 2.25 | 4.9.3 | 1.8.8 | *(none)* | 0.2.14 | 3.5.0 | 2.0.2 | 3.3.4 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2016a`` | Jan '16 | 2.25 | 4.9.3 | 1.10.2 | *(none)* | 0.2.15 | 3.6.0 | 2.0.2 | 3.3.4 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2016b`` | Jul '16 | 2.26 | 5.4.0 | 1.10.3 | *(none)* | 0.2.18 | 3.6.1 | 2.0.2 | 3.3.4 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2017a`` | Jan '17 | 2.27 | 6.3.0 | 2.0.2 | *(none)* | 0.2.19 | 3.7.0 | 2.0.2 | 3.3.6(-pl2) | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2017b`` | Jul '17 | 2.28 | 6.4.0 | 2.1.1 | *(none)* | 0.2.20* | (incl. with OpenBLAS) | 2.0.2 | 3.3.6(-pl2) | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2018a`` | Jan '18 | 2.28 | 6.4.0 | 2.1.2 | *(none)* | 0.2.20* | (incl. with OpenBLAS) | 2.0.2 | 3.3.7 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ -| ``2018b`` | Jul '18 | 2.30 | 7.3.0 | 3.1.1 | *(none)* | 0.3.1 | (incl. with OpenBLAS) | 2.0.2 | 3.3.8 | -+----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+ - -.. _common_toolchains_overview_intel_deprecated: - -Component versions in ``intel`` toolchain (deprecated versions) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``intel`` | *date* | *binutils* | *GCC* | *Intel compilers* | *Intel MPI* | *Intel MKL* | -+================+=============+=============+===========+====================+=============+==============+ -| ``2014b`` | Jul '14 | '*(none)* | 4.8.3 | 2013.5.192 | 4.1.3.049 | 11.1.2.144 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2015a`` | Jan '15 | '*(none)* | 4.9.2 | 2015.1.133 | 5.0.2.044 | 11.2.1.133 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2015b`` | Jul '15 | 2.25 | 4.9.3 | 2015.3.187 | 5.0.3.048 | 11.2.3.187 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2016a`` | Jan '16 | 2.26 | 4.9.3 | 2016.1.150 | 5.1.2.150 | 11.3.1.150 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2016b`` | Jul '16 | 2.26 | 5.4.0 | 2016.3.210 | 5.1.3.181 | 11.3.3.210 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2017a`` | Jan '17 | 2.27 | 6.3.0 | 2017.1.132 | 2017.1.132 | 2017.1.132 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2017b`` | Jul '17 | 2.28 | 6.4.0 | 2017.4.196 | 2017.3.196 | 2017.3.196 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2018a`` | Jan '18 | 2.28 | 6.4.0 | 2018.1.163 | 2018.1.163 | 2018.1.163 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ -| ``2018b`` | Jul '18 | 2.30 | 7.3.0 | 2018.3.222 | 2018.3.222 | 2018.3.222 | -+----------------+-------------+-------------+-----------+--------------------+-------------+--------------+ - -.. _common_toolchains_customizing: - -Customizing common toolchains ------------------------------ - - -Sometimes the need arises to customize one or more components of a common toolchain w.r.t. site-specific aspects. -One common example is using additional configuration options for Open MPI. - -To customize a toolchain component, you should copy the corresponding easyconfig file, -modify according to your needs, and make sure it is included in a location in the robot search path -that precedes the location of the easyconfig files that are included with EasyBuild -(see also :ref:`robot_search_path`), before building and installation the toolchain. - -More information about toolchains ---------------------------------- - -Please see the :ref:`toolchains_table` for how to obtain a listing of the currently known toolchains. - -For a detailed listing of the compiler options available with each toolchain, please see :ref:`avail_toolchain_opts`. diff --git a/docs/Concepts_and_Terminology.rst b/docs/Concepts_and_Terminology.rst deleted file mode 100644 index 541e977c..00000000 --- a/docs/Concepts_and_Terminology.rst +++ /dev/null @@ -1,187 +0,0 @@ -.. _concepts_and_terminology: - -Concepts and terminology -======================== - - -EasyBuild consists of a collection of Python modules and packages that interact with each other, -dynamically picking up additional Python modules as needed for building and installing -a (stack of) software package(s) specified via simple specification files. - -Or, in EasyBuild terminology: **the EasyBuild framework leverages easyblocks to automatically -build and install software using particular compiler toolchains, as specified by one or multiple easyconfig files**. - -.. contents:: - :depth: 3 - :backlinks: none - -.. _framework: - -EasyBuild framework -------------------- - -The EasyBuild **framework** embodies the core of the tool, providing functionality commonly -needed when installing scientific software on HPC systems. For example, it deals with downloading, -unpacking and patching of sources, loading module files for dependencies, -setting up the build environment, autonomously running (interactive) shell commands, -creating module files that match the specification files, etc. - -Included in the framework is an `abstract` implementation of a software build and install procedure, -which is split up into different `steps`: - -* unpacking sources -* configuration -* build -* installation -* module generation -* etc. - -Most of these steps, i.e., the ones which are generally more-or-less -analogous across different software packages, have appropriate (default) implementations. -The only exceptions are the configuration, build and installation steps that are purposely -left unimplemented (since there is no common procedure for them). - -Each of the steps can be -tweaked and steered via different parameters known to the framework, for which values are -either obtained from the provided specification files or set to reasonable default values. -See :ref:`easyconfig_files`. - -.. XXX - UPDATE BY VERSION FIXME - -In EasyBuild version |version| the framework source code consists of about 19000 lines of code, -organized across about 125 Python modules in roughly a dozen Python package directories, -next to almost 7000 lines of code for tests. This provides some notion of the size of the -EasyBuild framework and the amount of supporting functionality it has to offer. - - -.. _Easyblocks: - -Easyblocks ----------- - -The implementation of a particular software build and install procedure is done in a Python module, -which is aptly referred to as an **easyblock**. - -Each easyblock ties in with the framework API -by defining (or extending/replacing) one or more of the step functions that are part -of the abstract procedure used by the EasyBuild framework. Easyblocks typically heavily -rely on the supporting functionality provided by the framework, for example for -(autonomously) executing (interactive) shell commands and obtaining the command's output and exit code. - -A distinction is made between **software-specific** and **generic** easyblocks. Software-specific -easyblocks implement a build and install procedure which is entirely custom to one particular -software package (e.g., WRF), while generic easyblocks implement a procedure using standard -tools (e.g., CMake). Since easyblocks are implemented in an object-oriented scheme, the step -methods implemented by a particular easyblock can be reused in others via inheritance, -enabling code reuse across build procedure implementations. - -For each software package being built, the EasyBuild framework will determine which easyblock -should be used, based on the name of the software package or the value of the ``easyblock`` -specification parameter (see :ref:`writing_easyconfigs_easyblock_spec`). -Since EasyBuild v2.0, an easyblock *must* be specified in case no matching easyblock is found based on the -software name (cfr. :ref:`depr_ConfigureMake_fallback_eb1`). - -.. XXX - UPDATE BY VERSION FIXME - -EasyBuild version 2.4.0 includes 154 software-specific easyblocks and 28 generic -easyblocks (see also :ref:`basic_usage_easyblocks`), providing support for automatically installing a wide range -of software packages. Examples range from fairly easy-to-build programs like gzip, other basic tools -like compilers, various MPI stacks and commonly used libraries, primarily for x86_64 architecture systems, -to large scientific software packages that are notorious for their involved and tedious install procedures, such as: -`CP2K`, `NWChem`, `OpenFOAM`, `QuantumESPRESSO`, `WRF`. - -.. _toolchains: - -Toolchains ----------- - -EasyBuild employs so-called **compiler toolchains** or, simply `toolchains` for short, -which are a major concept in handling the build and installation processes. - -A typical toolchain consists of one or more compilers, usually put together with some libraries for specific functionality, -e.g., for using an MPI stack for distributed computing, or which provide optimized routines for commonly -used math operations, e.g., the well-known BLAS/LAPACK APIs for linear algebra routines. - -For each software package being built, the toolchain to be used must be specified in some way. - -The EasyBuild framework prepares the `build environment` for the different toolchain components, -by loading their respective modules and defining environment variables to specify compiler commands -(e.g., via ``$F90``), compiler and linker options (e.g., via ``$CFLAGS`` and ``$LDFLAGS``), the list -of library names to supply to the linker (via ``$LIBS``), etc. This enables making easyblocks largely -`toolchain-agnostic` since they can simply rely on these environment variables; that is, unless they -need to be aware of, for example, the particular compiler being used to determine the build configuration options. - -Recent releases of EasyBuild include out-of-the-box toolchain support for: - -- various compilers, including GCC, Intel, Clang, CUDA -- common MPI libraries, such as Intel MPI, MPICH, MVAPICH2, OpenMPI -- various numerical libraries, including ATLAS, Intel MKL, OpenBLAS, ScalaPACK, FFTW - -Please see the :ref:`common_toolchains` page for details about the two most common toolchains, -one for "free and open source software" (``foss``) based on GCC and one based on the Intel compilers -(``intel``). - -.. _system_toolchain: - -``system`` toolchain -~~~~~~~~~~~~~~~~~~~ - -The ``system`` toolchain is a special case. It is an *empty* toolchain, i.e. a toolchain without any components, -and corresponds to using the readily available compilers and libraries (e.g., the ones provided by the operating -system, or by modules which were loaded before issuing the ``eb`` command). - -When the ``system`` toolchain is used, a corresponding ``system`` module file is not required/loaded and no build -environment is being defined. - - -.. _dummy_toolchain: - -``dummy`` toolchain *(DEPRECATED)* -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The ``dummy`` toolchain has been deprecated in EasyBuild v4.0, and replaced by the :ref:`system_toolchain`. - - -Common toolchains -~~~~~~~~~~~~~~~~~ - -For more information on the concept of *common toolchains*, see :ref:`common_toolchains`. - -.. _easyconfig_files: - -Easyconfig files ----------------- - -The specification files that are supplied to EasyBuild are referred to as **easyconfig files** -(or simply `easyconfigs`), which are basically plain text files containing (mostly) -key-value assignments for build parameters supported by the framework, also referred -to as **easyconfig parameters** (see :doc:`Writing_easyconfig_files` for more information). - -Note that easyconfig files only provide the bits of information required -to determine the corresponding module name; the module name itself is computed by EasyBuild -framework by querying the module naming scheme being used. The complete -list of supported easyconfig parameters can be easily obtained via the EasyBuild command line using -``eb -a`` (see also :ref:`avail_easyconfig_params`). - -As such, each easyconfig file provides a complete specification of which particular software -package should be installed, and which settings should be used for building it. After completing -an installation, EasyBuild copies the used easyconfig file to the install directory, as a template, -and also supports maintaining an easyconfig archive which is updated on every successful installation. -Therefore, reproducing installations becomes trivial. - -.. _extensions: - -Extensions ----------- - -Some software packages support installing additional add-ons alongside the 'main' software, either in the same -installation prefix, or in a separate location. - -In EasyBuild, we use the neutral term '**extensions**' to refer these add-ons. - -Well-known examples include: - -* Perl modules (http://www.cpan.org/modules/) -* Python packages (https://pypi.python.org/pypi) -* R libraries (http://cran.r-project.org/web/packages/) -* Ruby gems (http://guides.rubygems.org/what-is-a-gem/) diff --git a/docs/Configuration.rst b/docs/Configuration.rst deleted file mode 100644 index a971b821..00000000 --- a/docs/Configuration.rst +++ /dev/null @@ -1,731 +0,0 @@ -.. _configuring_easybuild: - -Configuring EasyBuild -===================== - -This page discusses the recommended style of configuring -EasyBuild, which is supported since EasyBuild v1.3.0. - -A demo on configuring EasyBuild is available :ref:`here `. - -.. contents:: - :depth: 3 - :backlinks: none - -.. _configuration_types: - -Supported configuration types ------------------------------ - -Configuring EasyBuild can be done by: - -* using ``eb`` with **command line arguments** -* setting **environment variables** (``$EASYBUILD_...``) -* providing one or more **configuration files** - -Of course, combining any of these types of configuration works too (and -is even fairly common). - -The order of preference for the different configuration types is as listed above, that is: - -* environment variables override the corresponding entries in the configuration file -* command line arguments in turn override the corresponding environment variables *and* matching entries in the - configuration file - - -.. _configuration_consistency: - -Consistency across supported configuration types -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Note that the various available configuration options are handled -**consistently** across the supported configuration types. - -For example: to configure EasyBuild to use Lmod as modules tool, the following alternatives are available: - -* configuration file entry (key-value assignment): - - .. code:: ini - - [config] - modules-tool = Lmod - -* environment variable (upper case, ``EASYBUILD_`` prefix, ``-``'s becomes ``_``'s): - - .. code:: - - $ export EASYBUILD_MODULES_TOOL=Lmod - -* command line argument (long options preceded by ``--`` and (optionally) using ``=``): - - .. code:: - - $ eb --modules-tool=Lmod - - or - - .. code:: - - $ eb --modules-tool Lmod - -For more details w.r.t. each of the supported configuration types, see below. - - -.. _configuration_file: - -Configuration file(s) -~~~~~~~~~~~~~~~~~~~~~ - -.. _list_of_configuration_files: - -List of used configuration files -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The list of configuration files that will be used by EasyBuild is determined in the following order of preference: - -* the path(s) specified via the **command line argument** ``--configfiles`` -* the path(s) specified via the ``$EASYBUILD_CONFIGFILES`` **environment variable** -* the **default paths** for EasyBuild configuration files - -.. _default_configuration_files: - -Default configuration files -+++++++++++++++++++++++++++ - -By default, EasyBuild will use existing configuration files at the following paths: - -* ``$dir/easybuild.d/*.cfg``, for each directory ``$dir`` listed in ``$XDG_CONFIG_DIRS`` (where ``$XDG_CONFIG_DIRS`` - defaults to ``/etc``) -* ``$XDG_CONFIG_HOME/easybuild/config.cfg`` (where ``$XDG_CONFIG_HOME`` defaults to ``$HOME/.config``) - -Hence, if ``$XDG_CONFIG_HOME`` and ``$XDG_CONFIG_DIRS`` are not defined, EasyBuild will only consider default -configuration files at ``/etc/easybuild.d/*.cfg`` and ``$HOME/.config/easybuild/config.cfg``. - -The configuration file located in ``$XDG_CONFIG_HOME`` will be listed *after* the ones obtained via ``$XDG_CONFIG_DIRS``, -such that user-defined configuration settings have preference over system defaults. - -A detailed overview of the list of default configuration files is available via ``eb --show-default-configfiles`` -(available since EasyBuild v2.1.0). For example:: - - $ XDG_CONFIG_DIRS=/tmp/etc:/tmp/moreetc eb --show-default-configfiles - Default list of configuration files: - - [with $XDG_CONFIG_HOME: (not set), $XDG_CONFIG_DIRS: /tmp/etc:/tmp/moreetc] - - * user-level: ${XDG_CONFIG_HOME:-$HOME/.config}/easybuild/config.cfg - -> /home/example/.config/easybuild/config.cfg => found - * system-level: ${XDG_CONFIG_DIRS:-/etc}/easybuild.d/*.cfg - -> {/tmp/etc, /tmp/moreetc}/easybuild.d/*.cfg => /tmp/etc/easybuild.d/config.cfg, /tmp/moreetc/easybuild.d/bar.cfg, /tmp/moreetc/easybuild.d/foo.cfg - - Default list of existing configuration files (4): /tmp/etc/easybuild.d/config.cfg, /tmp/moreetc/easybuild.d/bar.cfg, /tmp/moreetc/easybuild.d/foo.cfg, /home/example/.config/easybuild/config.cfg - -Multiple configuration files -++++++++++++++++++++++++++++ - -If multiple configuration files are listed via a mechanism listed above, the settings in the last -configuration file have preference over the others. - -Each available configuration file will be used, and the configuration settings specified in these files -will be retained according to the order of preference as indicated above: settings in configuration files specified via -``--configfiles`` override those in configuration files specified via ``$EASYBUILD_CONFIGFILES``, which in turns override -settings in default configuration files. - -Ignored configuration files -+++++++++++++++++++++++++++ - -On top of this, the ``--ignoreconfigfiles`` configuration option allows to specify configuration files that should be -*ignored* by EasyBuild (regardless of whether they are specified via any of the options above). - - -Configuration file format -^^^^^^^^^^^^^^^^^^^^^^^^^ - -The EasyBuild configuration file follows the default Python -configuration format as parsed by the ``configparser`` module -(see `http://docs.python.org/2/library/configparser.html`_). - -Configuration files are organized in sections, the section name for a -particular configuration setting is indicated in the output of ``eb --help``. -Some examples sections are: ``MAIN``, ``basic``, ``config``, -``informative``, ``override``, ``regtest``, ``software``, ``unittest``, etc. - -Sections are indicated by specifying the section name in square brackets -on a dedicated line, e.g., ``[basic]``. - -Configuration settings are specified in a ``key = value`` or -``key: value`` format, **without using quotes for string-like values**. -For boolean configuration settings, values that evaluated to ``True`` -(e.g., ``true``, ``1``, …) are all equivalent to enabling the setting. - -Comment lines start with a hash character ``#`` (just like in Python code). - -An example configuration file that should make everything clear is shown below. - -.. code:: python - - [basic] - # always enable logging to stdout - logtostdout = true - [config] - # use Lmod as modules tool - modules-tool: Lmod - # use different default installation path - prefix=/home/you/work/easybuild/ - -.. _configuration_file_templates_constants: - -Templates and constants supported in configuration files -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Two types of template values ``%(...)s`` are supported in configuration files: - -* for configuration options defined in the configuration file (and only those) - - * *syntax:* ``%(opt)s``, i.e., using the (lowercase) name of the configuration option - -* for the default value of selected configuration options (see ``eb --avail-cfgfile-constants``) - - * *syntax:* ``%(DEFAULT_OPT)s``, i.e., using the uppercase name of the configuration option and prefixed with ``DEFAULT_`` - - -.. note:: - These template values are only supported in configuration files, *not* in environment variable values or - command line option values. - -.. note:: - Using an unknown template value, i.e. either one for a configuration option that was not defined in the - configuration file, or a non-existing one for a particular default value, will result in an error like: - ``ConfigParser.InterpolationMissingOptionError: Bad value substitution``. - -Example -+++++++ - -To include both the (custom) location for the easyconfigs archive repository and the default list of robot search -paths in the active robot search path, the following configuration file entry can be used, featuring the template -for the ``repositorypath`` configuration option and the provided ``DEFAULT_ROBOT_PATHS`` constant:: - - [basic] - repositorypath = /home/example/easybuild/easyconfigs_archive - robot-paths = %(repositorypath)s:%(DEFAULT_ROBOT_PATHS)s - -See also :ref:`controlling_robot_search_path`. - -Generating a template configuration file -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Since EasyBuild v1.10, a command line option ``--confighelp`` is -available that prints out the help text as an annotated configuration -file. This can be used as an empty template configuration file: - -.. code:: shell-session - - $ mkdir -p $HOME/.config/easybuild - $ eb --confighelp > $HOME/.config/easybuild/config.cfg - -.. code:: shell-session - - $ head $HOME/.easybuild/config.cfg - [MAIN] - # Enable debug log mode (def False) - #debug= - # Enable info log mode (def False) - #info= - # Enable info quiet/warning mode (def False) - #quiet= - - [basic] - # Print build overview incl. dependencies (full paths) (def False) - -.. _configuration_env_vars: - -Environment variables -~~~~~~~~~~~~~~~~~~~~~ - -All configuration settings listed as long options in ``eb --help`` can -also be specified via ``EASYBUILD_``-prefixed environment variables. - -Configuration settings specified this way always override the -corresponding setting specified in a configuration file. - -For example, to enable debug logging using an environment variable: - -.. code:: shell-session - - $ export EASYBUILD_DEBUG=1 - -More examples of using environment variables to configure EasyBuild are -shown in the sections below. - -.. tip:: Any configuration option of EasyBuild which can be tuned by command line - or via the configuration file, can also be tuned via a corresponding environment variable. - -.. note:: If any ``$EASYBUILD``-prefixed environment variables are defined that do not correspond to a known - configuration option, EasyBuild will report an error message and exit. - -.. _configuration_cmdline: - -Command line arguments -~~~~~~~~~~~~~~~~~~~~~~ - -The configuration type with the highest precedence are the ``eb`` -command line arguments, which override settings specified through environment variables or in configuration files. - -For some configuration options, both short and long command line -arguments are available (see ``eb --help``); the long options indicate -how the configuration setting should be specified in a configuration file -or via an environment variable (``$EASYBUILD_``). - -For boolean configuration settings, both the ``--