Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make outdir a mandatory parameter #1431

Merged
merged 9 commits into from
Mar 9, 2022
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/create-test-wf.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
- name: Run nf-core/tools
run: |
nf-core --log-file log.txt create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface"
nextflow run nf-core-testpipeline -profile test,docker
nextflow run nf-core-testpipeline -profile test,docker --outdir ./results

- name: Upload log file artifact
if: ${{ always() }}
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
* Add `.yamllint.yml` to avoid line length and document start errors ([#1407](https://github.com/nf-core/tools/issues/1407))
* Add `--publish_dir_mode` back into the pipeline template ([nf-core/rnaseq#752](https://github.com/nf-core/rnaseq/issues/752#issuecomment-1039451607))
* Add optional loading of of pipeline-specific institutional configs to `nextflow.config`
* Make `--outdir` a mandatory parameter ([nf-core/tools#1415](https://github.com/nf-core/tools/issues/1415))

## General

Expand Down
2 changes: 1 addition & 1 deletion nf_core/pipeline-template/.github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Learn more about contributing: [CONTRIBUTING.md](https://github.com/{{ name }}/t
- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/{{ name }}/tree/master/.github/CONTRIBUTING.md)
- [ ] If necessary, also make a PR on the {{ name }} _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
- [ ] Make sure your code lints (`nf-core lint`).
- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker`).
- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker` --outdir <OUTDIR>`).
- [ ] Usage Documentation in `docs/usage.md` is updated.
- [ ] Output Documentation in `docs/output.md` is updated.
- [ ] `CHANGELOG.md` is updated.
Expand Down
2 changes: 2 additions & 0 deletions nf_core/pipeline-template/.github/workflows/awsfulltest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ jobs:
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
compute_env: ${{ secrets.TOWER_COMPUTE_ENV }}
pipeline: ${{ github.repository }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm...where did this come from again?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And should we add it to awstest.yml too?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw it here and yes, as you nicely spotted it should also be added to awstest.yml

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have lost track of where we are with these actions. @ewels I could swear we tested this stuff and had already updated the pipeline template? I'm confused.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Given this code is in rnaseq on master and the Actions seems to be working I think we are good here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines were deliberately removed and should not have been added back here. Sorry I missed the ping fellas..

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Context - removed in #1395

This is because the new version of the Tower CLI tool needs the fully qualified GitHub URL now, instead of user/repo (which is what github.repository gives). The nf-core/tower-action does this properly by default now, but by specifying these in the template here we overwrite those defaults and break it.

Fixed in #1488

revision: ${{ github.sha }}
workdir: s3://${{ secrets.AWS_S3_BUCKET }}{% endraw %}/work/{{ short_name }}/{% raw %}work-${{ github.sha }}{% endraw %}
parameters: |
{
Expand Down
2 changes: 2 additions & 0 deletions nf_core/pipeline-template/.github/workflows/awstest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ jobs:
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
compute_env: ${{ secrets.TOWER_COMPUTE_ENV }}
pipeline: ${{ github.repository }}
revision: ${{ github.sha }}
workdir: s3://${{ secrets.AWS_S3_BUCKET }}{% endraw %}/work/{{ short_name }}/{% raw %}work-${{ github.sha }}{% endraw %}
parameters: |
{
Expand Down
4 changes: 2 additions & 2 deletions nf_core/pipeline-template/.github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ env:

jobs:
test:
name: Run workflow tests
name: Run pipeline with test data
# Only run on push if this is the nf-core dev branch (merged PRs)
if: {% raw %}${{{% endraw %} github.event_name != 'push' || (github.event_name == 'push' && github.repository == '{{ name }}') {% raw %}}}{% endraw %}
runs-on: ubuntu-latest
Expand Down Expand Up @@ -49,4 +49,4 @@ jobs:
# For example: adding multiple test runs with different parameters
# Remember that you can parallelise this by using strategy.matrix
run: |
nextflow run ${GITHUB_WORKSPACE} -profile test,docker
nextflow run ${GITHUB_WORKSPACE} -profile test,docker --outdir ./results
4 changes: 2 additions & 2 deletions nf_core/pipeline-template/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ On release, automated continuous integration tests run the pipeline on a full-si
3. Download the pipeline and test it on a minimal dataset with a single command:

```console
nextflow run {{ name }} -profile test,YOURPROFILE
nextflow run {{ name }} -profile test,YOURPROFILE --outdir <OUTDIR>
```

Note that some form of configuration will be needed so that Nextflow knows how to fetch the required software. This is usually done in the form of a config profile (`YOURPROFILE` in the example command above). You can chain multiple config profiles in a comma-separated string.
Expand All @@ -55,7 +55,7 @@ On release, automated continuous integration tests run the pipeline on a full-si
<!-- TODO nf-core: Update the example "typical command" below used to run the pipeline -->

```console
nextflow run {{ name }} -profile <docker/singularity/podman/shifter/charliecloud/conda/institute> --input samplesheet.csv --genome GRCh37
nextflow run {{ name }} --input samplesheet.csv --outdir <OUTDIR> --genome GRCh37 -profile <docker/singularity/podman/shifter/charliecloud/conda/institute>
```

## Documentation
Expand Down
8 changes: 4 additions & 4 deletions nf_core/pipeline-template/conf/modules.config
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@
Config file for defining DSL2 per module options and publishing paths
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Available keys to override module options:
ext.args = Additional arguments appended to command in module.
ext.args2 = Second set of arguments appended to command in module (multi-tool modules).
ext.args3 = Third set of arguments appended to command in module (multi-tool modules).
ext.prefix = File name prefix for output files.
ext.args = Additional arguments appended to command in module.
ext.args2 = Second set of arguments appended to command in module (multi-tool modules).
ext.args3 = Third set of arguments appended to command in module (multi-tool modules).
ext.prefix = File name prefix for output files.
----------------------------------------------------------------------------------------
*/

Expand Down
2 changes: 1 addition & 1 deletion nf_core/pipeline-template/conf/test.config
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
Defines input files and everything required to run a fast and simple pipeline test.

Use as follows:
nextflow run {{ name }} -profile test,<docker/singularity>
nextflow run {{ name }} -profile test,<docker/singularity> --outdir <OUTDIR>

----------------------------------------------------------------------------------------
*/
Expand Down
2 changes: 1 addition & 1 deletion nf_core/pipeline-template/conf/test_full.config
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
Defines input files and everything required to run a full size pipeline test.

Use as follows:
nextflow run {{ name }} -profile test_full,<docker/singularity>
nextflow run {{ name }} -profile test_full,<docker/singularity> --outdir <OUTDIR>

----------------------------------------------------------------------------------------
*/
Expand Down
2 changes: 1 addition & 1 deletion nf_core/pipeline-template/docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ An [example samplesheet](../assets/samplesheet.csv) has been provided with the p
The typical command for running the pipeline is as follows:

```console
nextflow run {{ name }} --input samplesheet.csv --genome GRCh37 -profile docker
nextflow run {{ name }} --input samplesheet.csv --outdir <OUTDIR> --genome GRCh37 -profile docker
```

This will launch the pipeline with the `docker` configuration profile. See below for more information about profiles.
Expand Down
4 changes: 2 additions & 2 deletions nf_core/pipeline-template/nextflow.config
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ params {
max_multiqc_email_size = '25.MB'

// Boilerplate options
outdir = './results'
outdir = null
tracedir = "${params.outdir}/pipeline_info"
publish_dir_mode = 'copy'
email = null
Expand Down Expand Up @@ -63,7 +63,7 @@ try {
System.err.println("WARNING: Could not load nf-core/config profiles: ${params.custom_config_base}/nfcore_custom.config")
}

// Load {{ name }} custom profiles from different institutions.
// Load {{ name }} custom profiles from different institutions.
// Warning: Uncomment only if a pipeline-specific instititutional config already exists on nf-core/configs!
// try {
// includeConfig "${params.custom_config_base}/pipeline/{{ short_name }}.config"
Expand Down
7 changes: 4 additions & 3 deletions nf_core/pipeline-template/nextflow_schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@
"fa_icon": "fas fa-terminal",
"description": "Define where the pipeline should find input data and save output data.",
"required": [
"input"
"input",
"outdir"
],
"properties": {
"input": {
Expand All @@ -26,8 +27,8 @@
},
"outdir": {
"type": "string",
"description": "Path to the output directory where the results will be saved.",
"default": "./results",
"format": "directory-path",
"description": "The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure.",
"fa_icon": "fas fa-folder-open"
},
"email": {
Expand Down
1 change: 1 addition & 0 deletions nf_core/pipeline-template/workflows/pipeline.nf
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ for (param in checkPathParamList) { if (param) { file(param, checkIfExists: true

// Check mandatory parameters
if (params.input) { ch_input = file(params.input) } else { exit 1, 'Input samplesheet not specified!' }
if (!params.outdir) { exit 1, 'Parameter outdir not specified!' }
drpatelh marked this conversation as resolved.
Show resolved Hide resolved

/*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand Down