question on building robust conditional workflow

[tamuanand]

Hi all

I am pretty new to NextFlow and I have been entrusted with the task of building a robust conditional workflow with several possibilities of software choice at each step. I have asked a question on the NextFlow google group - My original question is here on the NextFlow google group

To clarify: I am not trying to repost here. On this repo, I just happened to see something close to the situation I describe - use of HISAT2 or STAR as an aligner and I wanted to know more info.

My Scenario: Let me use RNASEQ-DEG analysis as an analogy

1. TrimGalore or Fastp or BBDUK
2. Kallisto or Salmon
3. EdgeR or DESEQ

If I chose TrimGalore in Step 1, there are 4 possible paths:

1. TrimGalore/Kallisto/EdgeR
2. TrimGalore/Kallisto/DESEQ
3. TrimGalore/Salmon/EdgeR
4. TrimGalore/Salmon/DESEQ

Hence, for the above scenario, there are 12 different paths available from start to finish depending on the combination of tools used at each step.

On this repo, I see this particular code block and I have some questions related to it

{
            "name": "aligner",
            "label": "Alignment tool",
            "usage": "Choose whether to align reads with STAR or HISAT2",
            "type": "string",
            "render": "radio-button",
            "choices": [
                    "star",
                    "hisat2"
            ],
            "default_value": "star",
            "group": "Alignment"
        },

Questions:

1. Is there an upstream webform that captures the radio button parameters and generates this json file?
2. How are the alternate options (radio button selections) implemented in your actual workflow

I was thinking of having an upstream web form that captures the input parameters from the user and use those parameters to call relevant processes from the NextFlow config.

I would appreciate any tips, best practices for the above.

Thanks in advance.

[ewels]

Hi @tamuanand,

Good question, and great digging skills! 😀 What you've found is one of our early pipeline schemas; this is an nf-core initiative (not nextflow, yet at least). It's still in an early stage, so not all nf-core pipelines have this file yet and the tools to use it are not very well developed. In other words, it may change in the future...

The file is a JSON Schema based on a main nf-core definition here. Currently, it is built entirely manually - in the future it would be nice to build helper tools to make this easier, but that hasn't happened yet. The intention for this file is to use it in a few different places:

• In the pipeline itself, to generate usage documentation (when --help is specified)
• In the pipeline, for self-validation - eg. preventing pipeline launch if required params are missing
• For development automated testing
  • The schema is validated, but we'd like to extend this to check that it matches what's in the pipeline script. Maybe even that all parameters are documented - stuff like that.
  • x-ref Add parameters.settings.json to template and linting tools#267
• In a command line wizard for launching pipelines
  • This part is done! ✨ https://nf-co.re/tools#launch-a-pipeline
• In a GUI - web based or application. This part is not done yet.
  • x-ref long thread here: Generic GUI for nf-core pipelines tools#61

So in answer to your questions:

1. No, there is no upstream webform yet. Though this would be relatively easily to build as we are using the JSON Schema standard. I'm interested in building something simple for this in the near future... 🤔
2. They're not implemented as such. Where you see "name": "aligner", this should match a params.aligner cli flag for the pipeline. Pipeline logic then typically expects radio values, but the JSON itself isn't used for this (yet).

Ok, I hope that's clear! So in summary an upstream web form is pretty much exactly the kind of thing that we're aiming for this. A generic solution may happen at some point, but if I'm honest I expect that the JSON file will be used by different people in different ways as this could be quite linked to local infrastructure needs.

Whilst the parameters JSON stuff is new and subject to change, I don't imagine that this file will disappear anywhere soon. So I think you're fine to build stuff using it.

Phil

[tamuanand]

Phil

A big Thank You for a very detailed response. Yes, you have answered my questions pertaining to webform/automation, etc. I will have to piece together different sections from different examples to get what I want.

Slightly off-topic, I see that there is a 4 day NextFlow training/camp in Barcelona in Sep 2019 and you are one of the speakers for the NextFlow Camp.

Question: Are there NextFlow training/workshops planned in the US/Canada too - if not it would be great to have one in US/Canada.

[ewels]

Great!

I will have to piece together different sections from different examples to get what I want.

Remember that the nf-core community is here to help, primarily on Slack. You can get an invite here. I suspect that quite a few of us will be interested in how you get on! 😉

Question: Are there NextFlow training/workshops planned in the US/Canada too - if not it would be great to have one in US/Canada.

That's a question for @pditommaso - Nextflow training is now run through http://seqera.io

[tamuanand]

Thanks Phil for the info.

@pditommaso I am based in the USA - pretty new to NextFlow and would like to move some of the bioinformatics workflows to NextFlow. Is it possible to do an online registration for the 4 day NextFlow training/camp in Barcelona in Sep 2019 so that I can join via Skype? Thanks.