Make batch jobs scale on failure

[jashapiro]

Looking at some of the code on nextflow-core for inspiration & best practices, I noticed that they often use code for memory usage declarations in the config file that looks something like this:

https://github.com/nf-core/rnaseq/blob/3b6df9bd104927298fcdf69e97cca7ff1f80527c/conf/base.config#L15

The idea being that if the first attempt fails for OOM, add memory and try again.

We can also set some error handling to make this specific to OOM errors and limit the number of retries

  errorStrategy = { task.exitStatus in [143,137,104,134,139] ? 'retry' : 'terminate' }
  maxRetries = 1

https://github.com/nf-core/rnaseq/blob/3b6df9bd104927298fcdf69e97cca7ff1f80527c/conf/base.config#L18-L19

We also have the ability to tag individual processes by resource requirements, like this, which seems like it will be a good idea:

 withLabel: low_memory {
    memory = { check_max( 16.GB * task.attempt, 'memory' ) }
  }
  withLabel: mid_memory {
    cpus = { check_max (4, 'cpus')}
    memory = { check_max( 28.GB * task.attempt, 'memory' ) }
    time = { check_max( 8.h * task.attempt, 'time' ) }
  }
  withLabel: high_memory {
    cpus = { check_max (10, 'cpus')}
    memory = { check_max( 70.GB * task.attempt, 'memory' ) }
    time = { check_max( 8.h * task.attempt, 'time' ) }
  }

https://github.com/nf-core/rnaseq/blob/3b6df9bd104927298fcdf69e97cca7ff1f80527c/conf/base.config#L23-L35

checkmax() is defined here: https://github.com/nf-core/rnaseq/blob/3b6df9bd104927298fcdf69e97cca7ff1f80527c/nextflow.config#L170-L172

[jashapiro]

An initial implementation of this appears in #17, but this should be moved to the config file and label implementation before closing.

[jashapiro]

This is now in the config file, but not all workflows use it. Standardizing this should happen before closing this issue!