add support for using 'sources' and 'git_config' for extensions in 'exts_list'

[kelseymh]

Modify fetch_extension_sources() so that when a source dictionary is included in an exts_list entry, it looks for git_config, and passes the latter through to obtain_file(). This allows us to pull extensions from Git the same way we pull regular sources, including creating a local tar-ball on the fly.

I plan to make this a "draft" (WIP) PR initially, because I haven't been able to explicitly test my code changes.

edit (@boegel): fixes #3251

[kelseymh]

I've successfully tested this feature with one of our internal Git packages. In my EasyConfig file, I have an extensions block:

exts_defaultclass = 'PythonPackage'
exts_list = [
    ('pyCAP', 'master', {
        'filename': 'pyCAP-master.tar.gz',
        'git_config': {
            'url': local_git + '/Analysis',
            'repo_name': 'pyCAP',
            'tag': 'master',
        }
    }),
]

When I run with the production eb (EasyBuild/2020a-sprint), I get the expected failure:

== fetching files...
== FAILED: Installation ended unsuccessfully (build directory: /scratch/group/mi
tchcomp/eb/tmp/build/cdmsPyGlobal/0.5/system-system-Python-3.6.6test):
build failed (first 300 chars): Couldn't find file pyCAP-master.tar.gz anywhere,
and downloading it didn't work either... Paths attempted (in order): /scratch/gr
oup/mitchcomp/eb/eb-files/c/cdmsPyGlobal/extensions/pyCAP-master.tar.gz, /scratc
h/group/mitchcomp/eb/eb-files/c/cdmsPyGlobal/packages/pyCAP-master.tar.gz, /scra
tch/group/m (took 0 sec)

but when I run with my personal checkout, it succeeds:

== fetching files...
== creating build dir, resetting environment...
[...]
== taking care of extensions...
== installing extension pyCAP master (1/1)...
[...]

Is there documentation for how to create a new unit test? I'd love to exercise this in some way as part of the pull request.

[kelseymh]

@boegel Suggested that a unit test could be added to either the easyblock.py or toy_build.py tests.

I think I can do this, but neither one appears to actually perform a download of sources. What I would want to test is a dictionary entry like

exts_list = [('toy', '1.0', {
    'filename':'toy-build.tar.gz', 
    'git_config': {
        'url':'https://github.com/something-something',
        'repo_name':'toy', 'tag':'1.0',
    }
})]

and see that either the download actually occurs (ideal) or at least that the contents of 'git_config' is seen and is validated.

[kelseymh]

I don't think I need to actually " see that ... the download actually occurs (ideal)." The filetools.py tests already include test_get_source_tarball_from_git(). That exercises both the different components of git_config as well as the downloading procedure.

For this situation, I just need to test whether having a 'git_config' dictionary in exts_list does not generate an error. This should be easier for me to figure out how to implement.

Update. Well, that didn't go so well. The test failed (which I expected), but then spewed hundreds of lines of DEBUG output. I'm sure an explanation for the failure is in there somewhere...

[kelseymh]

I just noticed that test_obtain_file() in the easyblock.py test does not exercise the git_config option. Should I try to add that as a unit test as well?

[akesandgren]

@boegel @kelseymh Shouldn't the format for this be

exts_list = [
  ('pyCAP', 'master', {
    'sources': [{
      'filename':...,
      'git_config'
    }],
  }),
]

To make it clear that it's the source specification, and make it behave like non-ext-list sources, allowing multiple sources files...

[kelseymh]

@akesandgren That's a good question! The allowed content of exts_list is undocumented, so I constructed that example block by trial and error.

In the code of this PR, I didn't want to try to change how exts_list parsed the extra_options dictionary, just added extraction of a git_config element.

The EasyConfigs which make use of 'sources' in a dictionary all seem to have it either at the top level or in default_component_specs (see, e.g., SpliceMap).

My write up of exts_list behaviour (see Framework PR #617 follows this searc of existing examples. If putting a whole 'sources': dictionary into extra_options does work, then I should update that attempt at documentation accordingly.

[boegel]

@kelseymh Great job on figuring this out, I'm sure that took a fair amount of effort...

An enhanced test for obtain_file to cover the use of git_config makes sense too, but that can also be tackled in a separate PR I think, up to you...

I agree with @akesandgren's remark w.r.t. git_config vs sources (only read his comment after I wrote up mine, so I'll leave it in place).

[kelseymh]

@boegel wrote:

An enhanced test for obtain_file to cover the use of git_config makes sense too, but that can also be tackled in a separate PR I think, up to you...

No need. I got most of the way through writing it, and discovered that test/framework/filetools.py already contains test_test_get_source_tarball_from_git(). I've closed the issue (#3322) I created for this.

[boegel]

Close & re-open to try and get Travis to wake up... You're running on fumes here Travis, seriously...