Version 1.1 merge Master > Dev

.travis.yml

@@ -6,6 +6,9 @@ python:
   - '3.4'
   - '3.5'
   - '3.6'
+before_install:
+  # PRs made to 'master' branch should always orginate from another repo or the 'dev' branch
+  - '[ $TRAVIS_PULL_REQUEST = "false" ] || [ $TRAVIS_BRANCH != "master" ] || ([ $TRAVIS_PULL_REQUEST_SLUG = $TRAVIS_REPO_SLUG ] && [ $TRAVIS_PULL_REQUEST_BRANCH = "dev" ])'
 install:
   # Install Nextflow
   - mkdir /tmp/nextflow
@@ -17,7 +20,10 @@ install:
   - pip install .
   - pip install codecov pytest pytest-datafiles pytest-cov mock
 
-script: python -m pytest --cov=nf_core .
+script:
+  - python -m pytest --cov=nf_core .
+  - nf-core create -n testpipeline -d "This pipeline is for testing"
+  - nf-core lint nf-core-testpipeline
 
 after_success:
   - codecov

CHANGELOG.md

@@ -1,8 +1,23 @@
 # nf-core/tools
 
-## v1.1dev
+## [v1.1](https://github.com/nf-core/tools/releases/tag/1.1) - 2018-08-14
+Very large release containing lots of work from the first nf-core hackathon, held in SciLifeLab Stockholm.
+
+* The [Cookiecutter template](https://github.com/nf-core/cookiecutter) has been merged into tools
+    * The old repo above has been archived
+    * New pipelines are now created using the command `nf-core create`
+    * The nf-core template and associated linting are now controlled under the same version system
+* Large number of template updates and associated linting changes
+    * New simplified cookicutter variable usage
+    * Refactored documentation - simplified and reduced duplication
+    * Better `manifest` variables instead of `params` for pipeline name and version
+    * New integrated nextflow version checking
+    * Updated travis docker pull command to use tagging to allow release tests to pass
+    * Reverted Docker and Singularity syntax to use `ENV` hack again
 * Improved Python readme parsing for PyPI
-* Update linting and release tools to support new style of Docker & Singularity conda installations
+* Updated Travis tests to check that the correct `dev` branch is being targeted
+* New sync tool to automate pipeline updates
+    * Once initial merges are complete, a nf-core bot account will create PRs for future template updates
 
 ## [v1.0.1](https://github.com/nf-core/tools/releases/tag/1.0.1) - 2018-07-18
 

MANIFEST.in

@@ -1,2 +1,3 @@
 include LICENSE
 include README.md
+recursive-include nf_core *

README.md

@@ -110,6 +110,46 @@ nf-core-methylseq-1.0/
 7 directories, 8 files
 ```
 
+## Creating a new workflow
+The `create` subcommand makes a new workflow using the nf-core base template. With a given pipeline name and description, it gives you a starter pipeline which follows nf-core best practices.
+
+After creating the files, the command initialises the folder as a git repository and makes an initial commit. This first "vanilla" commit which is identical to the output from the templating tool is important, as it allows us to keep your pipeline in sync with the base template in the future.
+
+```
+$ nf-core create -n nextbigthing -d "This pipeline analyses data from the next big 'omics technique"
+
+                                          ,--./,-.
+          ___     __   __   __   ___     /,-._.--~\
+    |\ | |__  __ /  ` /  \ |__) |__         }  {
+    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
+                                          `._,._,'
+
+
+INFO: Creating new nf-core pipeline: nextbigthing
+
+INFO: Initialising pipeline git repository
+
+INFO: Done. Remember to add a remote and push to GitHub!
+```
+
+Once you have run the command, create a new empty repository on GitHub under your username (not the `nf-core` organisation, yet).
+On your computer, add this repository as a git remote and push to it:
+
+```bash
+git remote add origin https://github.com/ewels/nf-core-nextbigthing.git
+git push --set-upstream origin master
+```
+
+You can then continue to edit, commit and push normally as you build your pipeline.
+When you're ready, create a new repository under the `nf-core` organisation (or ask someone to do this for you on the gitter channel) and make a pull-request.
+
+Final tasks (needs more documentation):
+* Set up travis CI on fork and nf-core repository
+* Create a dockerhub repository
+* Create a singularity hub repository
+* Add a description and keywords to the github repositories
+* Protect the `master` branch on the nf-core repository
+
 ## Linting a workflow
 The `lint` subcommand checks a given pipeline for all nf-core community guidelines.
 This is the same test that is used on the automated continuous integration tests.
@@ -186,12 +226,4 @@ INFO: Updating version in Singularity
 INFO: Updating version in environment.yml
  - name: nfcore-methylseq-1.3dev
  + name: nfcore-methylseq-1.3
-
-INFO: Updating version in Dockerfile
- - ENV PATH /opt/conda/envs/nfcore-methylseq-1.3dev/bin:$PATH
- + ENV PATH /opt/conda/envs/nfcore-methylseq-1.3/bin:$PATH
-
-INFO: Updating version in Singularity
- - PATH=/opt/conda/envs/nfcore-methylseq-1.3dev/bin:$PATH
- + PATH=/opt/conda/envs/nfcore-methylseq-1.3/bin:$PATH
 ```

bin/sync

@@ -0,0 +1,98 @@
+#!/usr/bin/env python
+
+from cookiecutter.main import cookiecutter
+import git
+import json
+import os
+import requests
+from requests.auth import HTTPBasicAuth
+import shutil
+import sys
+import subprocess
+import tempfile
+import syncutils.template
+
+# Set the default nf-core pipeline template branch
+DEF_TEMPLATE_BRANCH = "TEMPLATE"
+# The GitHub base url or the nf-core project
+GH_BASE_URL = "https://{token}@github.com/nf-core/{pipeline}"
+# The JSON file is updated on every push event on the nf-core GitHub project
+NF_CORE_PIPELINE_INFO = "http://nf-co.re/pipelines.json"
+# The API endpoint for creating pull requests
+GITHUB_PR_URL_TEMPL = "https://api.github.com/repos/nf-core/{pipeline}/pulls"
+
+
+def create_pullrequest(pipeline, origin="dev", template="TEMPLATE", token="", user="nf-core"):
+    """Create a pull request to a base branch (default: dev),
+    from a head branch (default: TEMPLATE)
+
+    Returns: An instance of class requests.Response
+    """
+    content = {}
+    content['title'] = "Important pipeline nf-core update! (version {tag})".format(tag=os.environ['TRAVIS_TAG'])
+    content['body'] = "Some important changes have been made in the nf-core pipelines templates.\n" \
+    "Please make sure to merge this in ASAP and make a new minor release of your pipeline.\n\n" \
+    "Follow the link [nf-core/tools](https://github.com/nf-core/tools/releases/tag/{})".format(os.environ['TRAVIS_TAG'])
+    content['head'] = "{}".format(template)
+    content['base'] = origin
+    return requests.post(url=GITHUB_PR_URL_TEMPL.format(pipeline=pipeline),
+                         data=json.dumps(content),
+                         auth=HTTPBasicAuth(user, token))
+
+def main():
+    # Check that the commit event is a GitHub tag event
+    assert os.environ['TRAVIS_TAG']
+    assert os.environ['NF_CORE_BOT']
+
+    # Catch exceptions in lists, and list them at the end
+    sync_errors = []
+    pr_errors = []
+
+    # Get nf-core pipelines info
+    res = requests.get(NF_CORE_PIPELINE_INFO)
+    pipelines = json.loads(res.content).get('remote_workflows')
+    if not pipelines:
+        print("Pipeline information was empty!")
+
+    # TODO: Remove this line, once we go for production
+    pipelines = [{"name":"hlatyping"}] # just for testing
+
+    # Update the template branch of each pipeline repo
+    for pipeline in pipelines:
+        print("Update template branch for pipeline '{pipeline}'... ".format(pipeline=pipeline['name']))
+        try:
+            syncutils.template.NfcoreTemplate(
+                pipeline['name'],
+                branch=DEF_TEMPLATE_BRANCH,
+                repo_url=GH_BASE_URL.format(token=os.environ["NF_CORE_BOT"], pipeline=pipeline['name'])
+            ).sync()
+        except Exception as e:
+            sync_errors.append((pipeline['name'], e))
+
+    # Create a pull request from each template branch to the origin branch
+    for pipeline in pipelines:
+        print("Trying to open pull request for pipeline {}...".format(pipeline['name']))
+        response = create_pullrequest(pipeline['name'], os.environ["NF_CORE_BOT"])
+        if response.status_code != 201:
+            pr_errors.append((pipeline['name'], response.status_code, response.content))
+        else:
+            print("Created pull-request for pipeline \'{pipeline}\' successfully."
+                .format(pipeline=pipeline["name"]))
+
+    for pipeline, exception in sync_errors:
+        print("Sync for pipeline {name} failed.".format(pipeline))
+        print(exception)
+
+    for pipeline, return_code, content in pr_errors:
+        print("Pull-request for pipeline \'{pipeline}\' failed," 
+                " got return code {return_code}."
+                .format(pipeline=pipeline["name"], return_code=return_code))
+        print(content)
+
+    if pr_errors or sync_errors: sys.exit(1)
+
+    sys.exit(0)
+
+if __name__ == "__main__":
+    main()
+

bin/syncutils/template.py

@@ -0,0 +1,88 @@
+import tempfile
+import utils
+import git
+import os
+import shutil
+import sys
+from cookiecutter.main import cookiecutter
+
+import nf_core.create
+
+
+class NfcoreTemplate:
+    """Updates the template content of an nf-core pipeline in
+    its `TEMPLATE` branch.
+
+    Args: - pipeline: The pipeline name
+          - branch: The template branch name, default=`TEMPLATE`
+          - token: GitHub auth token
+    """
+    def __init__(self, pipeline, branch='master', repo_url=""):
+        """Basic constructor
+        """
+        self.pipeline = pipeline
+        self.repo_url = repo_url
+        self.branch = branch
+        self.tmpdir = tempfile.mkdtemp()
+        self.templatedir = tempfile.mkdtemp()
+        self.repo = git.Repo.clone_from(self.repo_url, self.tmpdir)
+        assert self.repo
+
+    def sync(self):
+        """Execute the template update.
+        """
+        context = self.context_from_nextflow(nf_project_dir=self.tmpdir)
+        self.update_child_template(self.templatedir, self.tmpdir, context=context)
+        self.commit_changes()
+        self.push_changes()
+
+    def context_from_nextflow(self, nf_project_dir):
+        """Fetch a Nextflow pipeline's config settings.
+
+        Returns: A cookiecutter-readable context (Python dictionary)
+        """
+        # Check if we are on "master" (main pipeline code)
+        if self.repo.active_branch is not "master":
+            self.repo.git.checkout("origin/master", b="master")
+
+        # Fetch the config variables from the Nextflow pipeline
+        config = utils.fetch_wf_config(wf_path=nf_project_dir)
+
+        # Checkout again to configured template branch
+        self.repo.git.checkout("origin/{branch}".format(branch=self.branch),
+            b="{branch}".format(branch=self.branch))
+
+        return utils.create_context(config)
+
+
+    def update_child_template(self, templatedir, target_dir, context=None):
+        """Apply the changes of the cookiecutter template
+        to the pipelines template branch.
+        """
+        # Clear the pipeline's template branch content
+        for f in os.listdir(self.tmpdir):
+            if f == ".git": continue
+            try:
+                shutil.rmtree(os.path.join(target_dir, f))
+            except:
+                os.remove(os.path.join(target_dir, f))
+
+        # Create the new template structure
+        nf_core.create.PipelineCreate(
+            name=context.get('pipeline_name'),
+            description=context.get('pipeline_short_description'),
+            new_version=context.get('version'),
+            no_git=True,
+            force=True,
+            outdir=templatedir
+        )
+
+    def commit_changes(self):
+        """Commits the changes of the new template to the current branch.
+        """
+        self.repo.git.add(A=True)
+        self.repo.index.commit("Update nf-core pipeline template.")
+
+    def push_changes(self):
+        self.repo.git.push()
+

bin/syncutils/utils.py

@@ -0,0 +1,40 @@
+import os
+import subprocess
+
+def fetch_wf_config(wf_path):
+    """
+    Use nextflow to retrieve the nf configuration variables from a workflow
+    """
+    config = dict()
+    # Call `nextflow config` and pipe stderr to /dev/null
+    try:
+        with open(os.devnull, 'w') as devnull:
+            nfconfig_raw = subprocess.check_output(['nextflow', 'config', '-flat', wf_path], stderr=devnull)
+    except subprocess.CalledProcessError as e:
+        raise AssertionError("`nextflow config` returned non-zero error code: %s,\n   %s", e.returncode, e.output)
+    else:
+        for l in nfconfig_raw.splitlines():
+            ul = l.decode()
+            k, v = ul.split(' = ', 1)
+            config[k] = v.replace("\'", "").replace("\"", "")
+    return config
+
+def create_context(config):
+    """Consumes a flat Nextflow config file and will create 
+    a context dictionary with information for the nf-core template creation.
+
+    Returns: A dictionary with: 
+        {
+            'pipeline_name': '<parsed_name>'
+            'pipeline_short_description': '<parsed_description>'
+            'version': '<parsed_version>'
+        }
+    """
+    context = {}
+    context["pipeline_name"] = config.get("manifest.name") if config.get("manifest.name") else get_name_from_url(config.get("manifest.homePage"))
+    context["pipeline_short_description"] = config.get("manifest.description") 
+    context["version"] = config.get("manifest.version") if config.get("manifest.version") else config.get("params.version")
+    return context
+
+def get_name_from_url(url):
+    return url.split("/")[-1] if url else ""

docs/lint_errors.md

@@ -57,15 +57,18 @@ names. This test fails or throws warnings if required variables are not set.
 
 The following variables fail the test if missing:
 
-* `params.version`
-    * The version of this pipeline. This should correspond to a [GitHub release](https://help.github.com/articles/creating-releases/).
-* `params.nf_required_version`
-    * The minimum version of Nextflow required to run the pipeline.
-    * This should correspond to the `NXF_VER` version tested by Travis.
 * `params.outdir`
     * A directory in which all pipeline results should be saved
+* `manifest.name`
+    * The pipeline name. Should begin with `nf-core/`
 * `manifest.description`
     * A description of the pipeline
+* `manifest.pipelineVersion`
+    * The version of this pipeline. This should correspond to a [GitHub release](https://help.github.com/articles/creating-releases/).
+* `manifest.nextflowVersion`
+    * The minimum version of Nextflow required to run the pipeline.
+    * Should `>=` a version number, eg. `>=0.31.0`
+    * This should correspond to the `NXF_VER` version tested by Travis.
 * `manifest.homePage`
     * The homepage for the pipeline. Should be the nf-core GitHub repository URL,
       so beginning with `https://github.com/nf-core/`
@@ -98,15 +101,15 @@ This test fails if the following happens:
 
 * `.travis.yml` does not contain the string `nf-core lint ${TRAVIS_BUILD_DIR}` under `script`
 * `.travis.yml` does not contain the string `docker pull <container>` under `before_install`
-    * Where `<container>` is fetched from `params.container` in the `nextflow.config` file
-* `.travis.yml` does not test the Nextflow version specified in the pipeline as `nf_required_version`
+    * Where `<container>` is fetched from `params.container` in the `nextflow.config` file, without the docker tag _(if we have the tag the tests fail when making a release)_
+* `.travis.yml` does not test the Nextflow version specified in the pipeline as `manifest.nextflowVersion`
     * This is expected in the `env` section of the config, eg:
     ```yaml
     env:
       - NXF_VER=0.27.0
       - NXF_VER=''
     ```
-    * At least one of these `NXF_VER` variables must match the `params.nf_required_version` version specified in the pipeline config
+    * At least one of these `NXF_VER` variables must match the `manifest.nextflowVersion` version specified in the pipeline config
     * Other variables can be specified on these lines as long as they are space separated.
 
 ## Error #6 - Repository `README.md` tests ## {#6}
@@ -143,9 +146,9 @@ if they are set.
 > These tests only run when your pipeline has a root file called `environment.yml`
 
 * The environment `name` must match the pipeline name and version
-    * The pipeline name is found from the Nextflow config `manifest.homePage`,
-      which assumes that the URL is in the format `github.com/nf-core/[pipeline-name]`
-    * Example: For `github.com/nf-core/test` version 1.4, the conda environment name should be `nfcore-test-1.4`
+    * The pipeline name is defined in the config variable `manifest.name`
+    * Replace the slash with a hyphen as environment names shouldn't contain that character
+    * Example: For `nf-core/test` version 1.4, the conda environment name should be `nf-core-test-1.4`
 
 Each dependency is checked using the [Anaconda API service](https://api.anaconda.org/docs).
 Dependency sublists are ignored with the exception of `- pip`: these packages are also checked