Merge branch 'dev' into raise-for-singularity-container-retrieval-fail

.github/CONTRIBUTING.md

@@ -71,6 +71,24 @@ Or [version control integration](https://pycqa.github.io/isort/docs/configuratio
 There is an automated CI check that runs when you open a pull-request to nf-core/tools that will fail if
 any code does not adhere to isort formatting.
 
+### pre-commit hooks
+
+This repository comes with [pre-commit](https://pre-commit.com/) hooks for black, isort and Prettier. pre-commit automatically runs checks before a commit is committed into the git history. If all checks pass, the commit is made, if files are changed by the pre-commit hooks, the user is informed and has to stage the changes and attempt the commit again.
+
+You can use the pre-commit hooks if you like, but you don't have to. The CI on Github will run the same checks as the tools installed with pre-commit. If the pre-commit checks pass, then the same checks in the CI will pass, too.
+
+You can install the pre-commit hooks into the development environment by running the following command in the root directory of the repository.
+
+```bash
+pre-commit install --install-hooks
+```
+
+You can also run all pre-commit hooks without making a commit:
+
+```bash
+pre-commit run --all
+```
+
 ## API Documentation
 
 We aim to write function docstrings according to the [Google Python style-guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings). These are used to automatically generate package documentation on the nf-core website using Sphinx.

.github/workflows/create-lint-wf.yml

@@ -80,9 +80,13 @@ jobs:
       - name: nf-core modules update
         run: nf-core --log-file log.txt modules update --dir nf-core-testpipeline --all --no-preview
 
+      # Remove TODO statements
+      - name: remove TODO
+        run: find nf-core-testpipeline -type f -exec sed -i '/TODO nf-core:/d' {} \;
+
       # Run nf-core linting
       - name: nf-core lint
-        run: nf-core --log-file log.txt lint --dir nf-core-testpipeline --fail-ignored
+        run: nf-core --log-file log.txt lint --dir nf-core-testpipeline --fail-ignored --fail-warned
 
       # Run the other nf-core commands
       - name: nf-core list
@@ -98,7 +102,7 @@ jobs:
         run: nf-core --log-file log.txt bump-version --dir nf-core-testpipeline/ 1.1
 
       - name: nf-core lint in release mode
-        run: nf-core --log-file log.txt lint --dir nf-core-testpipeline --fail-ignored --release
+        run: nf-core --log-file log.txt lint --dir nf-core-testpipeline --fail-ignored --fail-warned --release
 
       - name: nf-core modules install
         run: nf-core --log-file log.txt modules install fastqc --dir nf-core-testpipeline/ --force

.pre-commit-config.yaml

@@ -0,0 +1,19 @@
+repos:
+  - repo: https://github.com/psf/black
+    rev: 22.3.0
+    hooks:
+      - id: black
+        language_version: python3.10
+  - repo: https://github.com/pycqa/isort
+    rev: 5.10.1
+    hooks:
+      - id: isort
+  - repo: https://github.com/pre-commit/mirrors-prettier
+    rev: "v2.6.2"
+    hooks:
+      - id: prettier
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: "v4.3.0"
+    hooks:
+      - id: name-tests-test
+        args: [--pytest-test-first]

CHANGELOG.md

@@ -9,19 +9,20 @@
 ### Linting
 
 - Check that the `.prettierignore` file exists and that starts with the same content.
+- Add isort configuration and GitHub workflow ([#1538](https://github.com/nf-core/tools/pull/1538))
 
 ### General
 
 - Fix and improve broken test for Singularity container download [#1622](https://github.com/nf-core/tools/pull/1622).
+- Updated the package requirements to prevent defunct installations of nf-core [#1620](https://github.com/nf-core/tools/pull/1620)
+- Add `--fail-warned` flag to `nf-core lint` to make warnings fail [#1593](https://github.com/nf-core/tools/pull/1593)
+- Add `--fail-warned` flag to pipeline linting workflow [#1593](https://github.com/nf-core/tools/pull/1593)
 
 ### Modules
 
+- Add `--fix-version` flag to `nf-core modules lint` command to update modules to the latest version ([#1588](https://github.com/nf-core/tools/pull/1588))
 - Fix a bug in the regex extracting the version from biocontainers URLs [#1598](https://github.com/nf-core/tools/pull/1598)
 
-### Linting
-
-- Add isort configuration and GitHub workflow ([#1538](https://github.com/nf-core/tools/pull/1538))
-
 ## [v2.4.1 - Cobolt Koala Patch](https://github.com/nf-core/tools/releases/tag/2.4) - [2022-05-16]
 
 - Patch release to try to fix the template sync ([#1585](https://github.com/nf-core/tools/pull/1585))

README.md

@@ -2,7 +2,8 @@
 
 [![Python tests](https://github.com/nf-core/tools/workflows/Python%20tests/badge.svg?branch=master&event=push)](https://github.com/nf-core/tools/actions?query=workflow%3A%22Python+tests%22+branch%3Amaster)
 [![codecov](https://codecov.io/gh/nf-core/tools/branch/master/graph/badge.svg)](https://codecov.io/gh/nf-core/tools)
-[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![code style: prettier](https://img.shields.io/badge/code%20style-prettier-ff69b4.svg)](https://github.com/prettier/prettier)
 [![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
 
 [![install with Bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/recipes/nf-core/README.html)

nf_core/__main__.py

@@ -301,9 +301,10 @@ def create(name, description, author, version, no_git, force, outdir):
 @click.option("-k", "--key", type=str, metavar="<test>", multiple=True, help="Run only these lint tests")
 @click.option("-p", "--show-passed", is_flag=True, help="Show passing tests on the command line")
 @click.option("-i", "--fail-ignored", is_flag=True, help="Convert ignored tests to failures")
+@click.option("-w", "--fail-warned", is_flag=True, help="Convert warn tests to failures")
 @click.option("--markdown", type=str, metavar="<filename>", help="File to write linting results to (Markdown)")
 @click.option("--json", type=str, metavar="<filename>", help="File to write linting results to (JSON)")
-def lint(dir, release, fix, key, show_passed, fail_ignored, markdown, json):
+def lint(dir, release, fix, key, show_passed, fail_ignored, fail_warned, markdown, json):
     """
     Check pipeline code against nf-core guidelines.
 
@@ -325,7 +326,7 @@ def lint(dir, release, fix, key, show_passed, fail_ignored, markdown, json):
     # Run the lint tests!
     try:
         lint_obj, module_lint_obj = nf_core.lint.run_linting(
-            dir, release, fix, key, show_passed, fail_ignored, markdown, json
+            dir, release, fix, key, show_passed, fail_ignored, fail_warned, markdown, json
         )
         if len(lint_obj.failed) + len(module_lint_obj.failed) > 0:
             sys.exit(1)
@@ -595,7 +596,8 @@ def create_test_yml(ctx, tool, run_tests, output, force, no_prompts):
 @click.option("-a", "--all", is_flag=True, help="Run on all modules")
 @click.option("--local", is_flag=True, help="Run additional lint tests for local modules")
 @click.option("--passed", is_flag=True, help="Show passed tests")
-def lint(ctx, tool, dir, key, all, local, passed):
+@click.option("--fix-version", is_flag=True, help="Fix the module version if a newer version is available")
+def lint(ctx, tool, dir, key, all, local, passed, fix_version):
     """
     Lint one or more modules in a directory.
 
@@ -608,7 +610,15 @@ def lint(ctx, tool, dir, key, all, local, passed):
     try:
         module_lint = nf_core.modules.ModuleLint(dir=dir)
         module_lint.modules_repo = ctx.obj["modules_repo_obj"]
-        module_lint.lint(module=tool, key=key, all_modules=all, print_results=True, local=local, show_passed=passed)
+        module_lint.lint(
+            module=tool,
+            key=key,
+            all_modules=all,
+            print_results=True,
+            local=local,
+            show_passed=passed,
+            fix_version=fix_version,
+        )
         if len(module_lint.failed) > 0:
             sys.exit(1)
     except nf_core.modules.lint.ModuleLintException as e:

nf_core/lint/__init__.py

@@ -29,7 +29,15 @@
 
 
 def run_linting(
-    pipeline_dir, release_mode=False, fix=(), key=(), show_passed=False, fail_ignored=False, md_fn=None, json_fn=None
+    pipeline_dir,
+    release_mode=False,
+    fix=(),
+    key=(),
+    show_passed=False,
+    fail_ignored=False,
+    fail_warned=False,
+    md_fn=None,
+    json_fn=None,
 ):
     """Runs all nf-core linting checks on a given Nextflow pipeline project
     in either `release` mode or `normal` mode (default). Returns an object
@@ -62,7 +70,7 @@ def run_linting(
     # Create the lint object
     pipeline_keys = list(set(key).intersection(set(PipelineLint._get_all_lint_tests(release_mode)))) if key else []
 
-    lint_obj = PipelineLint(pipeline_dir, release_mode, fix, pipeline_keys, fail_ignored)
+    lint_obj = PipelineLint(pipeline_dir, release_mode, fix, pipeline_keys, fail_ignored, fail_warned)
 
     # Load the various pipeline configs
     lint_obj._load_lint_config()
@@ -167,7 +175,7 @@ class PipelineLint(nf_core.utils.Pipeline):
     from .template_strings import template_strings
     from .version_consistency import version_consistency
 
-    def __init__(self, wf_path, release_mode=False, fix=(), key=(), fail_ignored=False):
+    def __init__(self, wf_path, release_mode=False, fix=(), key=(), fail_ignored=False, fail_warned=False):
         """Initialise linting object"""
 
         # Initialise the parent object
@@ -179,6 +187,7 @@ def __init__(self, wf_path, release_mode=False, fix=(), key=(), fail_ignored=Fal
         self.lint_config = {}
         self.release_mode = release_mode
         self.fail_ignored = fail_ignored
+        self.fail_warned = fail_warned
         self.failed = []
         self.ignored = []
         self.fixed = []
@@ -314,7 +323,10 @@ def _lint_pipeline(self):
                 for test in test_results.get("fixed", []):
                     self.fixed.append((test_name, test))
                 for test in test_results.get("warned", []):
-                    self.warned.append((test_name, test))
+                    if self.fail_warned:
+                        self.failed.append((test_name, test))
+                    else:
+                        self.warned.append((test_name, test))
                 for test in test_results.get("failed", []):
                     self.failed.append((test_name, test))
                 if test_results.get("could_fix", False):

nf_core/modules/lint/__init__.py

@@ -101,7 +101,16 @@ def __init__(self, dir):
     def _get_all_lint_tests():
         return ["main_nf", "meta_yml", "module_todos", "module_deprecations"]
 
-    def lint(self, module=None, key=(), all_modules=False, print_results=True, show_passed=False, local=False):
+    def lint(
+        self,
+        module=None,
+        key=(),
+        all_modules=False,
+        print_results=True,
+        show_passed=False,
+        local=False,
+        fix_version=False,
+    ):
         """
         Lint all or one specific module
 
@@ -118,6 +127,7 @@ def lint(self, module=None, key=(), all_modules=False, print_results=True, show_
         :param module:          A specific module to lint
         :param print_results:   Whether to print the linting results
         :param show_passed:     Whether passed tests should be shown as well
+        :param fix_version:     Update the module version if a newer version is available
 
         :returns:               A ModuleLint object containing information of
                                 the passed, warned and failed tests
@@ -174,11 +184,11 @@ def lint(self, module=None, key=(), all_modules=False, print_results=True, show_
 
         # Lint local modules
         if local and len(local_modules) > 0:
-            self.lint_modules(local_modules, local=True)
+            self.lint_modules(local_modules, local=True, fix_version=fix_version)
 
         # Lint nf-core modules
         if len(nfcore_modules) > 0:
-            self.lint_modules(nfcore_modules, local=False)
+            self.lint_modules(nfcore_modules, local=False, fix_version=fix_version)
 
         if print_results:
             self._print_results(show_passed=show_passed)
@@ -282,19 +292,21 @@ def get_installed_modules(self):
 
         return local_modules, nfcore_modules
 
-    def lint_modules(self, modules, local=False):
+    def lint_modules(self, modules, local=False, fix_version=False):
         """
         Lint a list of modules
 
         Args:
             modules ([NFCoreModule]): A list of module objects
             local (boolean): Whether the list consist of local or nf-core modules
+            fix_version (boolean): Fix the module version if a newer version is available
         """
         progress_bar = rich.progress.Progress(
             "[bold blue]{task.description}",
             rich.progress.BarColumn(bar_width=None),
             "[magenta]{task.completed} of {task.total}[reset] » [bold yellow]{task.fields[test_name]}",
             transient=True,
+            console=console,
         )
         with progress_bar:
             lint_progress = progress_bar.add_task(
@@ -305,9 +317,9 @@ def lint_modules(self, modules, local=False):
 
             for mod in modules:
                 progress_bar.update(lint_progress, advance=1, test_name=mod.module_name)
-                self.lint_module(mod, local=local)
+                self.lint_module(mod, progress_bar, local=local, fix_version=fix_version)
 
-    def lint_module(self, mod, local=False):
+    def lint_module(self, mod, progress_bar, local=False, fix_version=False):
         """
         Perform linting on one module
 
@@ -326,14 +338,17 @@ def lint_module(self, mod, local=False):
 
         # Only check the main script in case of a local module
         if local:
-            self.main_nf(mod)
+            self.main_nf(mod, fix_version, progress_bar)
             self.passed += [LintResult(mod, *m) for m in mod.passed]
             self.warned += [LintResult(mod, *m) for m in (mod.warned + mod.failed)]
 
         # Otherwise run all the lint tests
         else:
             for test_name in self.lint_tests:
-                getattr(self, test_name)(mod)
+                if test_name == "main_nf":
+                    getattr(self, test_name)(mod, fix_version, progress_bar)
+                else:
+                    getattr(self, test_name)(mod)
 
             self.passed += [LintResult(mod, *m) for m in mod.passed]
             self.warned += [LintResult(mod, *m) for m in mod.warned]