Download: Improve container detection in configs, specifically the container parameter in process scope

[MatthiasZepper]

When downloading a pipeline with --container-system singularity , tools will search for the respective containers at two different places:

1. In the modules (DSL2 syntax)
2. In the configs (Process scope)

While the module parsing was quite advanced, the config parsing assumed plain container definitions, which doesn't hold true e.g. for differentialabundance 1.2.0:

Therefore, I have now broken up the function find_container_images(). The part for cleaning the raw DSL2 matches was moved to a new helper function rectify_raw_container_matches() that is now also applied to the config matches.

Just a draft for now, tests etc. will be added tomorrow.

PR checklist

• This comment contains a description of changes (with reason)
• CHANGELOG.md is updated
• If you've fixed a bug or added code that should be tested, add tests!
• Documentation in docs is updated

[codecov]

Codecov Report

Merging #2346 (cdf620e) into dev (1d7c39c) will increase coverage by 0.06%.
The diff coverage is 89.85%.

@@            Coverage Diff             @@
##              dev    #2346      +/-   ##
==========================================
+ Coverage   72.69%   72.76%   +0.06%     
==========================================
  Files          78       78              
  Lines        8864     8889      +25     
==========================================
+ Hits         6444     6468      +24     
- Misses       2420     2421       +1     

Impacted Files	Coverage Δ
nf_core/download.py	59.47% <89.85%> (+1.24%)	⬆️

[mirpedrol]

Would adding the newline solve #2338 ?

Suggested change

	module_container = re.findall(r"container\s*[\"\']([^\"]*)[\"\']", search_space, re.S)
	module_container = re.findall(r"container\s*[\"\']([^\"]*)[\"\']\n", search_space, re.S)

[mirpedrol]

Please note that I haven't tested the full code, only this regex with an example module :)

[MatthiasZepper]

Sadly, not because there are modules where the DSL2 is wrapped into curly braces, so the quote is followed by a closing bracket. There are also modules with escaped quotes inside unescaped quotes that need to be matched.

After many hours, I realized that capturing the quote and using exactly this instead of an either or notation [\"\'] is the only thing that prevented abridged matches.

 container_regex = re.compile(
                            r"container\s+[\s{}=$]*(?P<quote>[\'\"])(?P<param>(?:.(?!\1))*.?)\1[\s}]*",
                            re.DOTALL
                        )

local_module_findings = re.findall(container_regex, search_space)

I have now tested about 20+ pipelines in multiple revisions, and it seems that will work for all modules and configs in there. There are still some containers not picked up (e.g. in Sarek 2.5), but this is not due to the regex, but due to the downstream logic how to prioritize https:// direct download container addresses over those that would need to be pulled with Singularity.

Maybe we should in the long-term either have a nf-core container API or include a small sqlite-Database with curated container addresses for all revisions into tools?

[mirpedrol]

I've tested downloading differentialabundance locally, and it seems to work and download the overridden container 👍

[mirpedrol]

LGTM, thanks!!