Optional input arguments, default value and named arguments.

[vruano]

New feature

In order to increase reusability of processes and (sub)workflows across pipelines we need flexibility in terms of what
arguments are customized at each invocation. The most convenient syntax would include both:

1. defining default values for arguments as well as
2. the possibility of skipping arguments by naming those that we are giving values to explicitly.

Number (2) is compatible to continue support of positional arguments even with mixtures as is supported by other languages like R.

Usage scenario

Improve dramatically reusability of processes and workflows across pipelines

Very often the same tools are used over and over by different analyses pipelines. These pipeline may need to customize the same tool invocation in different way. Therefore in order to be able to reuse the same code the tool wrapping process/workflow has to be
a "common multiple" in terms of arguments provide forcing each pipeline to provide sensible defaults for those arguments that are not relevant to them.

Shift tool wrapper development to the source

Adding support for optional arguments would give tool providing projects the chance to produce their own Nextflow API/bindings to access their tools for the benefit of Nextflow users. They wouldn't need to guess how their tools are going to be used by third parties. They simply produce flexible wrappers that expose every possible argument where only required one need to be explicitly set.

(What's the main usage case and the deployment scenario addressed by this proposal)

Suggest implementation

The syntax for processes definition would be something similar to this:

process tabix {

  input:
        path(input_file)
        val(regions = []) // empty means whole genome
        val(csi = false)
        val(comment = "#")
        val(skip_lines = 0)
        ....
}

Then you can invoke it like so:

     tabix(vcf_files, ["chr1", "chr2", "chr3:10000-200000"])
     tabix(input_file = vcf_files, regions =  ["chr1", "chr2", "chr3:10000-200000"])
     tabix(vcf_files, comment = ";")
     tabix(comment = ";", vcf_files) // anonymous parameters are taken in their declaration order.

By adding a default value you are implicitly declaring the input 'optional' and so there is no need for the 'optional' tag/keyword.

[bentsherman]

Related to #1694 and #2257 . 100% agree that named/optional inputs with default arguments would make processes much more re-usable. These features are in our backlog.

[pditommaso]

Having a default value when it's not specified it's indeed a good idea. Likely it may be better to rely on the current optional parameters syntax.

[notestaff]

Also related to #2127 . If maps were supported as process input, and one could specify a default map value, and when calling a process could specify a map as input value that would get merged with the default map, then this could implement the functionality requested here.

[bentsherman]

Going to start working on this one. Right now I have the following syntax in mind:

process foo {
    input:
    val name
    val revision, take: 'revision', default: 'latest'
    val extra, take: 'extra', optional: true
}

workflow {
    foo( 'name', revision='58a9f5')
}

Some key observations:

• by default, inputs are positional (required, no default, no keyword)
• inputs with default value are optional by default
• positional inputs must precede keyword inputs in the process definition

[bentsherman]

If all goes well it should make this monstrosity much easier to use 😄

[vruano]

Thanks for looking into this @bentsherman.
I have a couple of questions with answer that might be obvious for those with more insight in groovy DSL:

1. is it necessary to have to indicate the parameter name with 'take', could it just be the variable name by default.
2. could the order between positional and keyword arguments be more flexible (even mixed like in R)

[bentsherman]

1. is it necessary to have to indicate the parameter name with 'take', could it just be the variable name by default.

Yes, because some input types don't have to have a name. For example, a path input can specify a literal name instead of a variable name, and a tuple actually can't be named, it is defined only in terms of its components.

2. could the order between positional and keyword arguments be more flexible (even mixed like in R)

I think we would use the same convention as Groovy, which I believe allows the same flexibility that you're talking about. So you should be able to order the named args however you want, in both the process def and invocation, as long as your positional args are ordered consistently.

[vruano]

1. is it necessary to have to indicate the parameter name with 'take', could it just be the variable name by default.

Yes, because some input types don't have to have a name. For example, a path input can specify a literal name instead of a variable name, and a tuple actually can't be named, it is defined only in terms of its components.

I agree that if there is no variable name like in path ('fastqc/*') 'take' would be required. To clarify when I asked whether is necessary to indicate the 'take' option I mean whether is compulsory to do if a variable name is available like in the example code that you provide. It seems unnecessarily repetitive to have to write val XXX, take: 'XXX' as more often than not the variable name and the keyword would be exactly the same.

Number 2 is what I was hoping to hear thanks.

[bentsherman]

There is an important difference though: val XXX is a positional input whereas val XXX, take: 'XXX' is a named input. I suppose we could have a shorthand like val XXX, take: '*'... doesn't seem like you're saving much effort there.

On a more practical note, I would estimate that the vast majority of process inputs out in the wild are paths or tuples, where such a shorthand isn't relevant.

[bentsherman]

Here's an interesting question regarding default values -- should the default value be applied if an input channel is provided when calling the process, but the channel is empty at runtime? That would basically save you from having to add an ifEmpty(...) to the input channel.

@ewels I would love to hear your thoughts on this one. I'm inclined to say no because it could lead to unexpected behavior, but I'm curious what your experience would tell you.

[vruano]

mmmm.... I guess that I feel the need to insist in not having to add the repetitive take XXX without breaking back compatibility...

what if any formal parameter that has a variable name is potentially a named one ('take' is implied) but you still have the option to be positional.

So processing a process invocation and map argument provides with parameter would go like this.

1. We match all named arguments to the corresponding parameter.
2. The remaining unnamed arguments (positional) are assigned to parameters that have not yet received a value using their declaration order regardless of whether they have a name or not.
3. We fail if there are outstanding unnamed arguments (too many) or formal parameters that do not have a default value (too few arguments).

Example:

process TheProcess {
     input:
        val input1 
        val input2
        path ('input3_*', take : 'input3')  // in this case is necessary to use take if you want it named.
        path 'input4_*' // this one remains unnamed.
        val input5
        val input6, default : 6
        val input7, default : 7
   ...
}

...
TheProcess(1, 2, path3, path4, 5, 6, 7) // all parameters assigned a value based on declaration order.
TheProcess(input3=path3, input5=5, 1, 2, path4, 6, 7) // same as above but using names to be more explicit.
TheProcess(1, 2, path4, 6, 7, input5 = 5, input3 = path3) // same; order and position of named arguments is irrelevant.
TheProcess(1, 2, path3, path4, 5) // input 6th and 7th use their default values.
TheProcess(1, 2, path3) // Error too few arguments
TheProcess(1, 2, path3, path4, 5, 6, 7, 8) // error too many arguments
TheProcess(input1 = 1, input1 = 11, 2, ...) // error same argument provided twice.
// ...etc...

In the design above, although not enforced, it would make more sense to push the parameters with default values to the bottom of the list.