CWL Viewer

This is a Spring Boot MVC application which fetches Common Workflow Language files from a Github repository and creates a page for it detailing the main workflow and its inputs, outputs and steps.

Using CWL Viewer

You are recommended to use the production instance of CWL Viewer at https://view.commonwl.org/ which runs the latest release. Any downtime should be reported on the gitter chat for cwlviewer.

Running

If you are a developer, or you want to use the CWL Viewer in a closed environment, then you can run your own instance.

Recommended - Running with Docker

This application can be started with Docker and Docker Compose and Docker Compose is the recommended method of running or developing this codebase.

Then run the following commands to clone the project in your local system.

 git clone https://github.com/common-workflow-language/cwlviewer.git
 cd cwlviewer

In the project directory, to start CWLViewer exposed on port 8080, run:

docker compose up

The web server will connect to a local host, you'll see the message saying "Tomcat started on port(s):8080".

To see the locally running CWL Viewer app, visit http://localhost:8080/ in your web browser.

To stop and remove:

docker compose down

If you change the source code, then use this docker-compose.override.yml and re-build with docker compose build:

version: '3.9'
services:
  spring:
    build: .

See the docker-compose.yml file for details.

If you have modified the source code, then you may want to build the docker image locally first:

docker build -t commonworkflowlanguage/cwlviewer .

Running Spring Boot locally for development, with PostgreSQL and Jena Fuseki in Docker

Create docker-compose.override.yml:

version: '3.9'
services:
  postgres:
    ports:
     - "5432:5432"
  sparql:
    ports:
     - "3030:3030"

Then start the containers:

docker compose up

Then start Spring Boot locally:

mvn spring-boot:run -Dserver.port=7999

Now you can connect to http://localhost:7999 in your browser.

Deleting the data volumes to reset state

To completely reset the state, you must delete the data volumes:

docker compose down
docker volume rm  cwlviewer_bundle cwlviewer_git cwlviewer_graphviz cwlviewer_postgres cwlviewer_sparql

Running without Docker

Requirements

PostgreSQL

You will need to have PostgreSQL running, by default on localhost:5432

If you are running from the command line, you can override this by supplying system properties like -Dspring.datasource.url=jdbc:postgresql://localhost:5432/cwlviewer and -Dspring.datasource.password=sa

Apache Jena Fuseki (or alternative SPARQL server)

You will also need to have a SPARQL server such as Apache Jena Fuseki running, by default on localhost:3030

Ruby and Licensee

To retrieve license information, CWL Viewer uses the Licensee Ruby Gem. To install it, configure Ruby on your environment and then run

gem install licensee

You may use a dependency from your operating system package manager if you prefer too, e.g. ruby-licensee for Ubuntu LTS 22.04.1.

Before running Maven, try running licensee in the command-line to verify it was installed successfully.

Compiling and Running

To compile you will need Java 17 or a compatible distribution (e.g. Eclipse Adoptium) and version, as well as Apache Maven 3 (apt install maven).

Spring Boot uses an embedded HTTP server. The Spring Boot Maven plugin includes a run goal which can be used to quickly compile and run it:

$ mvn spring-boot:run

Alternatively, you can run the application from your IDE as a simple Java application by importing the Maven project.

You need to install Graphviz for all unit tests to pass.

You can create an executable JAR file by using:

mvn clean install

Afterwards, run:

java -jar target/cwlviewer*.jar

(The exact filename will vary per version)

Once CWL Viewer is running, you should see log output somewhat like:

()..)
s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
org.researchobject.CwlViewerApplication  : Started CwlViewerApplication in 28.604 seconds

Now check out http://localhost:8080/ to access CWL Viewer.

Configuration

There are a variety of configuration options detailed in the application configuration file which can be adjusted.

When deploying with docker, these can be overridden externally by creating/modifying docker-compose.override.yml as follows:

version: '3.9'
services:
  spring:
    environment:
            applicationName: Common Workflow Language Viewer
            applicationURL: https://view.commonwl.org
            cacheDays: 1

The properties can alternatively be provided as system properties on the command line, e.g. -DcacheDays=1 or via a variety of other methods supported by Spring Boot

Dump/restore

While you can perform backup of the Docker volumes, for larger upgrades of CWL Viewer it is recommended instead to do a JSON dump and re-load, which will force CWL Viewer to fetch and parse again.

The script dump.py can be used for regular backups, it will store the full output of /workflows as one or multiple timestamped JSON files (you can use gzip to compress them):

$ python dump.py --viewer https://view.commonwl.org/ --output /var/backups --page 0 --size 100
  INFO:Viewer URL: https://view.commonwl.org/
  INFO:Output: /var/backups
  INFO:Dumping workflows from https://view.commonwl.org/, page 0, size 100 to /var/backups

$ python dump.py -o /var/backups -a
  INFO:Viewer URL: https://view.commonwl.org/
  INFO:Output: /var/backups
  INFO:Dumping all the workflows from https://view.commonwl.org/ to /var/backups
  100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [04:39<00:00, 17.49s/it]

The script load.py (requires Python 3) can be used to restore from such JSON dumps:

./load.py /var/backups/cwl/2018-06-06T135133+0000.json.gz https://view.commonwl.org/

The optional parameter --no-commits can be added to skip those entries that look like a commit ID. Note that this might break previous permalinks.

Documentation

2017 Poster https://doi.org/10.7490/f1000research.1114375.1?

2017 Video overview https://youtu.be/_yjhVTmvxLU

2017 Technical Report https://doi.org/10.5281/zenodo.823295

License

Distributed under the Apache License, Version 2.0. See the file LICENSE.md for details, and NOTICE.md for required attribution notices.

Contribute

Feel free to contribute! You may raise an issue, provide a pull request or join the gitter chat for cwlviewer!

Changelog

See CHANGELOG

Making a development snapshot container image

(and optionally publishing that image to DockerHub)

# confirm the build arguments
# if these don't look correct, troubleshoot before continuing.
BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') VCS_REF=$(git rev-parse HEAD) VERSION=$(git describe)
echo BUILD_DATE=${BUILD_DATE} VCS_REF=${VCS_REF} VERSION=${VERSION}
# build the container image
docker build --build-arg BUILD_DATE=${BUILD_DATE} --build-arg VCS_REF=${VCS_REF} \
  --build-arg VERSION=${VERSION} \
  -t cwlviewer:${VERSION} .
# the rest is optional
docker tag cwlviewer:${VERSION} docker.io/commonworkflowlanguage/cwlviewer:${VERSION}
docker tag cwlviewer:${VERSION} quay.io/commonwl/cwlviewer:${VERSION}
docker push docker.io/commonworkflowlanguage/cwlviewer:${VERSION}
docker push quay.io/commonwl/cwlviewer:${VERSION}

Making a release and publishing to GitHub, DockerHub, and Quay.io

After CHANGELOG.md has been updated and the -SNAPSHOT suffix removed from pom.xml, run the following:

git checkout main
git pull
new_version=1.4.3  # CHANGEME
# create an annotated git tag
git tag -a -m "release version ${new_version}" v${new_version}
# confirm the build arguments
# if these don't look correct, troubleshoot before continuing.
# for example, was your tag an annotated (-a) tag?
BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') VCS_REF=$(git rev-parse HEAD) VERSION=$(git describe)
echo BUILD_DATE=${BUILD_DATE} VCS_REF=${VCS_REF} VERSION=${VERSION}
# build the container image
docker build --build-arg BUILD_DATE=${BUILD_DATE} --build-arg VCS_REF=${VCS_REF} \
  --build-arg VERSION=${VERSION} \
  -t cwlviewer:${VERSION} .
# tag this container image in preparation for pushing to Docker Hub and Quay.io
docker tag cwlviewer:${VERSION} docker.io/commonworkflowlanguage/cwlviewer:${VERSION}
docker tag cwlviewer:${VERSION} docker.io/commonworkflowlanguage/cwlviewer:latest
docker tag cwlviewer:${VERSION} quay.io/commonwl/cwlviewer:${VERSION}
docker tag cwlviewer:${VERSION} quay.io/commonwl/cwlviewer:latest
# push the container image to Docker Hub and Quay.io
docker push docker.io/commonworkflowlanguage/cwlviewer:${VERSION}
docker push docker.io/commonworkflowlanguage/cwlviewer:latest
docker push quay.io/commonwl/cwlviewer:${VERSION}
docker push quay.io/commonwl/cwlviewer:latest
# upload the annotated tag to GitHub
git push --tags
git push

Then copy the changelog into https://github.com/common-workflow-language/cwlviewer/releases/new using the tag you just pushed.

Finally, make a new PR to bump the version and restore the -SNAPSHOT suffix in pom.xml.

Thanks

Developers and contributors include:

• Mark Robinson http://orcid.org/0000-0002-8184-7507
• Stian Soiland-Reyes http://orcid.org/0000-0001-9842-9718
• Michael Crusoe http://orcid.org/0000-0002-2961-9670
• Carole Goble http://orcid.org/0000-0003-1219-2137
• Charles Overbeck https://github.com/coverbeck
• Finn Bacall http://orcid.org/0000-0002-0048-3300
• Osakpolor Obaseki https://github.com/obasekiosa
• Bruno P. Kinoshita https://orcid.org/0000-0001-8250-4074

Thanks to:

• eScience Lab at The University of Manchester
• BioExcel Center of Excellence for Computational Biomolecular Research
• • European Commission's H2020 grant 675728
• Common Workflow Language community and the CWL Gitter room