Clinica's Pydra-based Engine

[omar-rifai]

The purpose of this entry is to list and discuss the design choices around Clinica's engine 2.0 (with Pydra).

Overall Architecture

The main idea is to delegate most of the work which is common across pipelines to the engine. For this, we want the role of the engine - given an arbitrary Pydra core workflow - to be threefold:

1. Make it BIDS-aware by appending an output workflow and prepending an input workflow
2. Add provenance: Extract provenance information and save it to disk
3. Convert to Clinica command line tool: Add it to the pipelines that can be executed with Clinica

Schematically, we can represent this threefold enhancement as follows:

BIDS awarness

In order to handle arbitrary workflows, it is necessary to define the I/O interfaces dynamically. Indeed, the input and output specifications of the input workflow and output workflow need to change to according to the specifications of the core workflow.

Provenance

Provenance information as specified in BEP028 needs to be added the the workflows. Activity level information should be extractable from Pydra's audit module but access to the I/O interfaces will allow us to customize the level of information tracked.

Command line tool

By converting an arbitrary workflow into a Clinica pipeline, we integrate it as a command line tool and allow users to use it though Clinica's interface. As such, the addition of parameters such as working dir, cache dir, command line arguments and name can be done at this level.

[ghisvail]

One possible implementation discussed using successive decorators, and their impact on the construction of the final pipeline.

Starting with a non-BIDS core workflow:

def core_workflow(*args) -> pydra.Workflow
    ...

flowchart LR
    CW(Core Workflow)

Loading

Making it BIDS-compatible:

@bidsify(...)  # returns pydra.Workflow
def core_workflow(*args) -> pydra.Workflow
    ...

flowchart LR
    CW(Core Workflow)
    IW(Input BIDS)
    OW(Output BIDS)

    IW --> CW --> OW

Loading

Add support for provenance:

@provify(...)  # returns pydra.Workflow
@bidsify(...)  # returns pydra.Workflow
def core_workflow(*args) -> pydra.Workflow
    ...

flowchart LR
    CW(Core Workflow)
    IW(Input BIDS)
    OW(Output BIDS)
    PW(Provenance)

    IW --> CW --> OW
    IW --> PW
    OW --> PW
    PW --> OW

Loading

Make it a Clinica pipeline:

@pipeline(...)  # returns clinica.Pipeline
@provify(...)   # returns pydra.Workflow
@bidsify(...)   # returns pydra.Workflow
def core_workflow(*args) -> pydra.Workflow
    ...

flowchart LR
    subgraph Pipeline
    CW(Core Workflow)
    IW(Input BIDS)
    OW(Output BIDS)
    PW(Provenance)

    IW --> CW --> OW
    IW --> PW
    OW --> PW
    PW --> OW
    end

Loading