add (experimental) support for specifying easyconfig files via "easystack" file

[deniskristak]

Added support for easybuild-ing from provided specsfile. Currently works with yaml, can be extended to json etc.
Original task: #3468

[boegel]

@deniskristak (very) quick feedback:

• I don't think there's a need to spread the new code you have across different files in easybuild/tools/specsfile/*.py , just add a single file easybuild/tools/specsfile.py with all of it (it'll still be short).
• clean it up a bit so the code style checks are happy (you can check yourself by running flake8, see https://flake8.pycqa.org).
• try to add tests for the functions you've implemented in specsfile.py

Maybe we also need a better name than "specsfile", which sounds very generic... Any suggestions on that?

cc @casparvl

[casparvl]

Nice start Denis! Just for fun, I tried to run it on the example yaml file I provided in my original issue and got

Bioconductor-3.11-foss-2020a

 full_path: []

Bioconductor does not have clearly specified parameters - 0 matches found. Skipping.

R-4.0.0-foss-2020a

 full_path: []

R does not have clearly specified parameters - 0 matches found. Skipping.

GROMACS-2020.1-foss-2020a

 full_path: []

GROMACS does not have clearly specified parameters - 0 matches found. Skipping.

GROMACS-2020.3-foss-2020a

 full_path: ['/hpc/eb/easyconfigs-surfsara/g/GROMACS/GROMACS-2020.3-foss-2020a.eb']

GROMACS-2020.1-fosscuda-2020a

 full_path: []

GROMACS does not have clearly specified parameters - 0 matches found. Skipping.

EasyBuild-4.3.1-SYSTEM-

 full_path: []

EasyBuild does not have clearly specified parameters - 0 matches found. Skipping.

OpenFOAM-8-foss-2020a

 full_path: ['/hpc/eb/easyconfigs-surfsara/o/OpenFOAM/OpenFOAM-8-foss-2020a.eb']

OpenFOAM-v2006-foss-2020a

 full_path: []

OpenFOAM does not have clearly specified parameters - 0 matches found. Skipping.


 determined_paths: ['/hpc/eb/easyconfigs-surfsara/g/GROMACS/GROMACS-2020.3-foss-2020a.eb', '/hpc/eb/easyconfigs-surfsara/o/OpenFOAM/OpenFOAM-8-foss-2020a.eb']


WARNING: Failed to determine toolchain hierarchy for foss/2020a when determining subtoolchain for dependency 'CMake': 'Could not find easyconfig for foss toolchain version 2020a'

looks like what you intended :)

Maybe we also need a better name than "specsfile", which sounds very generic... Any suggestions on that?

Hm, good one. Everything sounds unclear, or too long to me. Buildspec? Buildspecfile? Buildlist? Buildlistspec? The problem with the first two is that it doesn't reflect that this concerns (typically) building more than one software. "Buildlist" is again quite generic, and doesn't cover the level of detail with which you can specify build behaviour (once we allow e.g. specifying command line arguments per item in the yaml file). "Buildlistspec" covers quite well what the yaml file is, but may be a bit long. And I find all of my suggestions ugly, so I'm open to other suggestions too ;-)

I've checked a bit the code. And I have two major points

(1) I see that you're currently calling search_easyconfigs. I'm not sure if that is a wise road to go down: what if multiple EasyConfigs are found? Or none? How is EasyBuild supposed to continue from that? In my view, the specfile should specify unambiguously which EasyConfig is intended. I.e. to me

software:
 GROMACS:
    toolchains:
      foss-2020a:
        exclude-labels: system:gpu
        versions:
          2020.1:
            from_pr: 1234

Should quite litterally do the equivalent of

eb GROMACS-2020.1-foss-2020a.eb --from-pr 1234

since the specfile really specifies all (except the .eb extension) the information that is needed. A spec missing e.g. a toolchain version is simply an invalid spec and should just error out - just like e.g. eb GROMACS-2020.1-foss.eb would error out saying that that EasyConfig can't be found.

Currently, it does more of a

eb -S GROMACS-2020.1-foss-20202a

Grab the resulting full path (if a unique one is returned) and pass that on to eb.main's determined_path parameter.

Why not just parse the spec file to compose a new orig_paths and pass that back to main? The only reason I can imagine might be our request for supporting wildcards, which you may have intended to do through the search_eascyonfigs, but... I would probably do that at a more generic EB level so that a eb GROMACS-*-foss-2020a.eb would also just build all versions of gromacs with that toolchain that are available. In other words: I'd make det_easyconfig_paths more intelligent so that it can deal with wildcards - not try to implement that (only) for our specfile option.

(2) The way it is currently set up, a call to eb example.yaml would lead to a single call to EB's main, parse the YAML file, extract the determined_paths and then proceed. But: how will that enable us to allow specification of additional arguments (such as a --from-pr) for some easyconfigs, but not for others? I don't think that is possible: I think one call to EB's main will always result in one set of arguments that is applied to all EasyConfigs.

What I would do is probably: parse the yaml file, and compose new calls to eb.main from that (with appropriate arguments). That keeps the additional specfile option simpler (the only thing it needs to do is extract an EasyConfig name and related options, and create a loop over that parsed list that simply calls main again).

@boegel what do you think?

suggestions taken into account