Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strip params docs #700

Merged
merged 8 commits into from
Jul 30, 2020
Original file line number Diff line number Diff line change
@@ -1,55 +1,5 @@
# {{ cookiecutter.name }}: Usage

## Table of contents

* [Table of contents](#table-of-contents)
* [Introduction](#introduction)
* [Running the pipeline](#running-the-pipeline)
* [Updating the pipeline](#updating-the-pipeline)
* [Reproducibility](#reproducibility)
* [Main arguments](#main-arguments)
* [`-profile`](#-profile)
* [`--input`](#--input)
* [`--single_end`](#--single_end)
* [Reference genomes](#reference-genomes)
* [`--genome` (using iGenomes)](#--genome-using-igenomes)
* [`--fasta`](#--fasta)
* [`--igenomes_ignore`](#--igenomes_ignore)
* [Job resources](#job-resources)
* [Automatic resubmission](#automatic-resubmission)
* [Custom resource requests](#custom-resource-requests)
* [AWS Batch specific parameters](#aws-batch-specific-parameters)
* [`--awsqueue`](#--awsqueue)
* [`--awsregion`](#--awsregion)
* [`--awscli`](#--awscli)
* [Other command line parameters](#other-command-line-parameters)
* [`--outdir`](#--outdir)
* [`--publish_dir_mode`](#--publish_dir_mode)
* [`--email`](#--email)
* [`--email_on_fail`](#--email_on_fail)
* [`--max_multiqc_email_size`](#--max_multiqc_email_size)
* [`-name`](#-name)
* [`-resume`](#-resume)
* [`-c`](#-c)
* [`--custom_config_version`](#--custom_config_version)
* [`--custom_config_base`](#--custom_config_base)
* [`--max_memory`](#--max_memory)
* [`--max_time`](#--max_time)
* [`--max_cpus`](#--max_cpus)
* [`--plaintext_email`](#--plaintext_email)
* [`--monochrome_logs`](#--monochrome_logs)
* [`--multiqc_config`](#--multiqc_config)

## Introduction

Nextflow handles job submissions on SLURM or other environments, and supervises running the jobs. Thus the Nextflow process must run until the pipeline is finished. We recommend that you put the process running in the background through `screen` / `tmux` or similar tool. Alternatively you can run nextflow within a cluster job submitted your job scheduler.

It is recommended to limit the Nextflow Java virtual machines memory. We recommend adding the following line to your environment (typically in `~/.bashrc` or `~./bash_profile`):

```bash
NXF_OPTS='-Xms1g -Xmx4g'
```

<!-- TODO nf-core: Document required command line parameters to run the pipeline-->
ewels marked this conversation as resolved.
Show resolved Hide resolved

## Running the pipeline
Expand Down Expand Up @@ -130,7 +80,7 @@ You can also supply a run name to resume a specific run: `-resume [run-name]`. U

Specify the path to a specific config file (this is a core NextFlow command). See the [nf-core website documentation](https://nf-co.re/usage/configuration) for more information.

### Custom resource requests
#### Custom resource requests

Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with an error code of `143` (exceeded requested resources) it will automatically resubmit with higher requests (2 x original, then 3 x original). If it still fails after three times then the pipeline is stopped.

Expand All @@ -149,3 +99,21 @@ See the main [Nextflow documentation](https://www.nextflow.io/docs/latest/config
If you are likely to be running `nf-core` pipelines regularly it may be a good idea to request that your custom config file is uploaded to the `nf-core/configs` git repository. Before you do this please can you test that the config file works with your pipeline of choice using the `-c` parameter (see definition below). You can then create a pull request to the `nf-core/configs` repository with the addition of your config file, associated documentation file (see examples in [`nf-core/configs/docs`](https://github.com/nf-core/configs/tree/master/docs)), and amending [`nfcore_custom.config`](https://github.com/nf-core/configs/blob/master/nfcore_custom.config) to include your custom profile.

If you have any questions or issues please send us a message on [Slack](https://nf-co.re/join/slack) on the [`#configs` channel](https://nfcore.slack.com/channels/configs).

### `-bg`

Nextflow handles job submissions and supervises the running jobs. The Nextflow process must run until the pipeline is finished.

The Nextflow `-bg` flag launches Nextflow in the background, detached from your terminal so that the workflow does not stop if you log out of your session. The logs are saved to a file.

Alternatively, you can use `screen` / `tmux` or similar tool to create a detached session which you can log back into at a later time.
Some HPC setups also allow you to run nextflow within a cluster job submitted your job scheduler (from where it submits more jobs).

#### Nextflow memory requirements

In some cases, the Nextflow Java virtual machines can start to request a large amount of memory.
We recommend adding the following line to your environment to limit this (typically in `~/.bashrc` or `~./bash_profile`):

```bash
NXF_OPTS='-Xms1g -Xmx4g'
```