Oliver

An opinionated Cromwell orchestration manager.
Explore the docs »

Request Feature · Report Bug · ⭐ Consider starring the repo! ⭐

🎨 Features

• Workflow Orchestration. Easily submit, inspect, kill, and retry workflows in a Cromwell environment.
• Better Job Tracking. Jobs can be associated with names and job groups to enable better status reporting.
• Dynamic Argument Parsing. Specify inputs and options on the command line rather than editing JSON files.
• Third-party Cloud Integrations. Use the aws and azure subcommands to explore cloud-specific functionality.

📚 Getting Started

Installation

Conda

Oliver is distributed as a package using the community-curated Anaconda repository, conda-forge. You'll need to install conda, and we recommend that you first follow the instructions included in the conda-forge documentation to get everything set up!

conda install oliver -c conda-forge

Python Package Index

You can also install Oliver using the Python Package Index (PyPI).

pip install stjudecloud-oliver

Configuring

Next, we recommend that you configure oliver so that common arguments can be saved. By default, Oliver will prompt you for the answers interactively.

oliver configure

If you are setting up Oliver programmatically, you can accept a default configuration (oliver configure --defaults) and edit from there using oliver config.

🚌 A Quick Tour

At its foundation, Oliver is an opinionated job orchestrator for Cromwell. Commonly, you will want to use it to submit a job, inspect a job's status, kill a job, retry a job (possibly with different parameters), and organize job results.

If you're interested in a complete overview of Oliver's capabilities, please see the documentation pages.

Submit a Job

The simplest possible job submission is one which submits a simple workflow with one or more input JSON file(s) and/or key-value pair(s).

oliver submit workflow.wdl inputs.json input_key=input_value

You can similarly set workflow options and labels by prepending arguments with @ and % respectively.

# works for files too!
oliver submit workflow.wdl @option=foo %label=bar

Please see the docs for more details on job submission.

Inspect a Job

Once a job is submitted, you can interrogate the Cromwell server about its status.

oliver inspect workflow-id

If you aren't sure what workflow identifier was given to your job, you can easily track it down using the status subcommand.

# detailed view, which shows individual workflow statuses
oliver status -d

Kill a Job

If, for whatever reason, you'd like to stop a job, you can use Oliver to instruct Cromwell to do so.

oliver kill workflow-id

Retry a Job

Retrying a workflow is similarly easy: even if you need to override previously set parameters (e.g. increase the memory capacity for a task).

# override previous inputs by specifying arguments (the same way as you would for `submit`).
oliver retry workflow-id

🖥️ Development

If you are interested in contributing to the code, please first review our CONTRIBUTING.md document. To bootstrap a development environment, please use the following commands.

# Clone the repository
git clone git@github.com:stjudecloud/oliver.git
cd oliver

# Install the project using poetry
poetry install

# Ensure pre-commit is installed to automatically format
# code using `black`.
brew install pre-commit
pre-commit install
pre-commit install --hook-type commit-msg

🚧️ Tests

Oliver provides a (currently patchy) set of tests — both unit and end-to-end. To get started with testing, you'll need to bootstrap a Docker test environment (one-time operation).

# Start development environment
docker image build --tag oliver .
docker-compose up --build  -d

alias docker-run-oliver="docker container run \
  -it \
  --rm \
  --network oliver_default \
  --mount type=bind,source=$PWD/seeds,target=/opt/oliver/seeds \
  --mount type=bind,source=$PWD/oliver,target=/opt/oliver/oliver \
  --mount type=bind,source=$PWD/tests,target=/opt/oliver/tests \
  --entrypoint '' \
  oliver:latest"

# Seed development environment (make sure Cromwell is live first!)
docker-run-oliver bash seeds/seed.sh http://cromwell:8000 seeds/wdl/hello.wdl
docker-run-oliver pytest --cov=./ --cov-report=xml

To reset your entire docker-compose environment, you can run the following:

docker-compose down

docker image rm oliver:latest
docker image rm oliver_cromwell:latest
docker image rm mysql:5.7
docker volume rm oliver_mysql_data
docker network rm oliver_default

docker image build --tag oliver .
docker-compose up --build -d

🤝 Contributing

Contributions, issues and feature requests are welcome!
Feel free to check issues page. You can also take a look at the contributing guide.

📝 License

Copyright © 2020 St. Jude Cloud Team.
This project is MIT licensed.