Conditionals in Janis

[illusional]

Conditionals

Introducing "Conditionals" into Janis!

Essentially, we want to to decide at runtime whether to run a task (or not). This concept is based off how CWL, WDL and Nextflow's implementation of this feature. Essentially, we you can insert a condition into a when block, and if that condition is True at runtime, the step will execute.

We'll also add some extra sugar for some basic workflow concepts, such as "if else".

This isn't a fully fleshed out spec yet, and may change without notice:

Basic concept

• If you add a "when" block when adding a w.step. The outputs become optional.
• Follow on behaviour TBA:
  • Considering adding a "skip all future tasks if skipped" option
  • The output become optional, so provided your task can handle an optional input, you should be fine.

Basic syntax

w = WorkflowBuilder("conditionalTest")

w.input("inp", int, value=1)
w.input("name", str, value="Michael")

# I only want to run "echo" if the input called 'inp' has value greater than 1
w.step("echo", Echo(inp=w.name), when=w.inp > 1)

w.output("out", source=w.echo.out)

Conditional

Say you want to run an if, else if, else if block. Writing this by hand is a little complicated, and then you'd need to handle the merge yourself. Well, that's why we introduced the workflow.conditional. It allows you to write the equivalent statements, and Janis will take care of abstracting the logic.

w = WorkflowBuilder("ifelsetestTest")

w.input("inp", int, value=2)
w.input("inp1", str, value="Hello")
w.input("inp2", str, value="Hi there")

# Equivalent to:
#     if (inputs.inp > 1) { run Echo(inp=w.inp1 + " is greater than one") }
#     else if (inputs.inp == 1) { run Echo(inp="Input is equal to 1") }
#     else { run Echo(inp="The value " + w.inp2 + " is greater than one") }
w.conditional("echoconditional", [
    (w.inp > 1, Echo(inp=w.inp1 + " is greater than one")), 
    (w.inp == 1, Echo(inp="Input is equal to 1")),
    Echo(inp="The value " + w.inp2 + " is greater than one")
])

w.output("out", source=w.echoconditional)

Handling dependencies

If a step includes some conditionality (through the when block), all of it's outputs are marked as optional. The inputs of these dependent tools should be optional (where appropriate), otherwise a typing error should occur.

A SkipIfNull operator should be introduced to ensure that an input to a tool is valid. This should interact with the type check to not throw error, but this might actually be difficult to determine.

We could potentially include a skip_dependencies param on the initial workflow, which adds a skipIfNull condition onto all future steps down the tree that

Early exit

Proposing to add Workflow.error block, to early exit a workflow when a condition is met. It would be a commandlinetool with no outputs, so it couldn't be referenced anywhere.

wf.error("message", when=<condition>)

Handling multiple inputs

CWL introduces pickValue, where WDL would use the select_all and select_first library functions. The functionality can be mirrored:

:: CWL == WDL ::
all_non_null == select_all(X)
first_non_null == select_first(X)
only_non_null == {
	if (length(select_all(X)) > 1) throw Error
	return select_first(X)
}

I propose we introduce the function select_all and select_first, this aligns well with the proposal to introduce the standard library functions, and this PR makes good groundwork through operators. Could be compared to miniWDL's std.Lib in how extensible it could be.

CWL support

For CWL, we're modelling the behaviour off the most current PR for Conditionals (in CWL v1.2): common-workflow-language/cwl-v1.2#4.

As discussed, conditionals is only supported in v1.2 of CWL. I've been mostly mainting CWLGen, but due to the rise of CWLUtils #5, schema-salad has the functionality of generating Python classes that directly mirror the CWL spec. This means CWLGen can be deprecated, and we'll likely devise some "versioning" system for the translations, modelled off these new CWLUtils changes.

WDL support

WDL supports conditionality directly in version 1.0, (although we use the Development Biscayne spec). I've added WorkflowConditional to python-wdlgen in python-wdlgen/e656cc1.

Similar to CWL, I'd really prefer to build an AST (like in MiniWDL) and let something pretty print this to a string. Relevant:

• Conversation from Gitter: Jan 23 07:26
• Generate WDL from a parsed tree chanzuckerberg/miniwdl#216

[evan-wehi]

So that looks pretty useful. Is it only constrained to inputs or will it be possible to work on outputs such as stdout, return code, etc?

[illusional]

Hey @evan-wehi, you could definitely use the outputs of tasks to drive the conditional logic, that would include stdout if you explicitly include this.

The return code is a little trickier, in CWL v1.1 you're able to access the return code within an outputEval block with runtime.exitCode (so theoretically yes) for command line tools, we could automatically make Janis do this. Although it's not directly available in WDL, given that it's bash I'm sure we could make Janis collect it and pipe it back as an output rc of a tool.

[illusional]

This PR has been folded into #38, so see that for more information.