Add annotation + viola + join checks + cleanup (#29)

* add join checks

* fix delly mismatches

* Added viola

* fix header issues in single caller runs

* header fix?

* appease eclint

* add stub to reverse_bed

* fix for gridss?

* appease eclint again!

* typo

* fix permissions

* fix shebang

* fix shebang again

* disable gridss

* fix tests

* remove old code from gatk-sv

* Add VEP to the pipeline

* add some QOL parameters

* update tests

* QOL changes (variables and joins)

* disable scattering for delly

* remove unused gatk modules

* update modules

* made BED files optional

* add metro map and update readme

* forgot to save svg

* fix linting

* fix vep test

* free up CI space?

* use another cache for vep

.github/workflows/ci.yml

@@ -29,13 +29,20 @@ jobs:
           - "22.10.5"
           - "latest-everything"
         test:
-          - "all"
+          - "all_annotate"
+          - "all_no_annotate"
           - "smoove"
           - "delly"
           - "manta"
-          - "whamg"
-          - "gridss"
+
     steps:
+      - name: Free some space
+        run: |
+          sudo rm -rf "/usr/local/share/boost"
+          sudo rm -rf "$AGENT_TOOLSDIRECTORY"
+          sudo rm -rf /usr/share/dotnet
+          sudo rm -rf /opt/ghc
+
       - name: Check out pipeline code
         uses: actions/checkout@v3
 

README.md

@@ -8,22 +8,13 @@
 
 ## Introduction
 
-<!-- TODO nf-core: Write a 1-2 sentence summary of what data the pipeline is for and what it does -->
-
-**CenterForMedicalGeneticsGhent/nf-cmgg-structural** is a bioinformatics best-practice analysis pipeline for calling structural variants.
+**CenterForMedicalGeneticsGhent/nf-cmgg-structural** is a bioinformatics best-practice analysis pipeline for calling germline structural variants from short reads.
 
 The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html) implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from [nf-core/modules](https://github.com/nf-core/modules) in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community!
 
-<!-- TODO nf-core: Add full-sized test dataset and amend the paragraph below if applicable -->
-
-On release, automated continuous integration tests run the pipeline on a full-sized dataset on the AWS cloud infrastructure. This ensures that the pipeline runs on AWS, has sensible resource allocation defaults set to run on real-world datasets, and permits the persistent storage of results to benchmark between pipeline releases and other analysis sources.The results obtained from the full-sized test can be viewed on the [nf-core website](https://nf-co.re/centerformedicalgeneticsghent-nf-cmgg-structural/results).
-
 ## Pipeline summary
 
-<!-- TODO nf-core: Fill in short bullet-pointed list of the default steps in the pipeline -->
-
-1. Read QC ([`FastQC`](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/))
-2. Present QC for raw reads ([`MultiQC`](http://multiqc.info/))
+![metro map](docs/images/metro_map.png)
 
 ## Quick Start
 
@@ -46,30 +37,22 @@ On release, automated continuous integration tests run the pipeline on a full-si
 
 4. Start running your own analysis!
 
-   <!-- TODO nf-core: Update the example "typical command" below used to run the pipeline -->
-
    ```bash
-   nextflow run CenterForMedicalGeneticsGhent/nf-cmgg-structural --input samplesheet.csv --outdir <OUTDIR> --genome GRCh37 -profile <docker/singularity/podman/shifter/charliecloud/conda/institute>
+   nextflow run CenterForMedicalGeneticsGhent/nf-cmgg-structural --input samplesheet.csv --outdir <OUTDIR> --genome GRCh38 -profile <docker/singularity/podman/shifter/charliecloud/conda/institute>
    ```
 
 ## Documentation
 
-The nf-core/centerformedicalgeneticsghent-nf-cmgg-structural pipeline comes with documentation about the pipeline [usage](https://nf-co.re/centerformedicalgeneticsghent-nf-cmgg-structural/usage), [parameters](https://nf-co.re/centerformedicalgeneticsghent-nf-cmgg-structural/parameters) and [output](https://nf-co.re/centerformedicalgeneticsghent-nf-cmgg-structural/output).
+The CenterForMedicalGenetics/nf-cmgg-structural pipeline comes with documentation about the pipeline [usage](https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-structural/tree/master/docs/usage.md) and [output](https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-structural/tree/master/docs/output.md).
 
 ## Credits
 
 CenterForMedicalGeneticsGhent/nf-cmgg-structural was originally written by Nicolas Vannieuwkerke and Mattias Van Heetvelde.
 
-We thank the following people for their extensive assistance in the development of this pipeline:
-
-<!-- TODO nf-core: If applicable, make list of people who have also contributed -->
-
 ## Contributions and Support
 
 If you would like to contribute to this pipeline, please see the [contributing guidelines](.github/CONTRIBUTING.md).
 
-For further information or help, don't hesitate to get in touch on the [Slack `#centerformedicalgeneticsghent-nf-cmgg-structural` channel](https://nfcore.slack.com/channels/centerformedicalgeneticsghent-nf-cmgg-structural) (you can join with [this invite](https://nf-co.re/join/slack)).
-
 ## Citations
 
 <!-- TODO nf-core: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file. -->

assets/header.txt

@@ -154,6 +154,8 @@
 ##INFO=<ID=ALLVARS_EXT,Number=.,Type=String,Description="A comma-separated of all variants supporting this call">
 ##INFO=<ID=VARCALLS,Number=1,Type=String,Description="The number of variant calls supporting this variant">
 ##INFO=<ID=INTRASAMPLE_IDLIST,Number=.,Type=String,Description="The IDs which were merged in the most recent round of merging">
+##INFO=<ID=SVLENORG,Number=1,Type=Integer,Description="The SV length calculated by the used variant caller">
+##INFO=<ID=ORGBEID,Number=1,Type=String,Description="The original ID of the breakend">
 ##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
 ##FORMAT=<ID=GL,Number=G,Type=Float,Description="Log10-scaled genotype likelihoods for RR,RA,AA genotypes">
 ##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality">

assets/samplesheet.csv

@@ -1,6 +1,5 @@
 sample,family,cram,crai,bed
 PosCon1,family1,s3://test-data/genomics/homo_sapiens/illumina/cram/SVcontrol/small/PosCon1.cram,s3://test-data/genomics/homo_sapiens/illumina/cram/SVcontrol/small/PosCon1.cram.crai,s3://test-data/genomics/homo_sapiens/illumina/regions/SVcontrol/PosCon1and2.roi.bed
 PosCon2,family1,s3://test-data/genomics/homo_sapiens/illumina/cram/SVcontrol/small/PosCon2.cram,s3://test-data/genomics/homo_sapiens/illumina/cram/SVcontrol/small/PosCon2.cram.crai,s3://test-data/genomics/homo_sapiens/illumina/regions/SVcontrol/PosCon1and2.roi.bed
-PosCon3,,s3://test-data/genomics/homo_sapiens/illumina/cram/SVcontrol/small/PosCon3.cram,s3://test-data/genomics/homo_sapiens/illumina/cram/SVcontrol/small/PosCon3.cram.crai,s3://test-data/genomics/homo_sapiens/illumina/regions/SVcontrol/PosCon3.roi.bed
+PosCon3,,s3://test-data/genomics/homo_sapiens/illumina/cram/SVcontrol/small/PosCon3.cram,s3://test-data/genomics/homo_sapiens/illumina/cram/SVcontrol/small/PosCon3.cram.crai,
 PosCon4,,s3://test-data/genomics/homo_sapiens/illumina/cram/SVcontrol/small/PosCon4.cram,s3://test-data/genomics/homo_sapiens/illumina/cram/SVcontrol/small/PosCon4.cram.crai,s3://test-data/genomics/homo_sapiens/illumina/regions/SVcontrol/PosCon4.roi.bed
-PosCon5,,s3://test-data/genomics/homo_sapiens/illumina/cram/SVcontrol/small/PosCon5.cram,s3://test-data/genomics/homo_sapiens/illumina/cram/SVcontrol/small/PosCon5.cram.crai,s3://test-data/genomics/homo_sapiens/illumina/regions/SVcontrol/PosCon5.roi.bed

bin/simple-event-annotation.R

@@ -0,0 +1,55 @@
+#!/usr/local/bin/Rscript
+# Fetched from https://github.com/PapenfussLab/gridss/blob/master/example/simple-event-annotation.R
+# Although it's been slightly adjusted
+
+# if (!requireNamespace("BiocManager", quietly = TRUE)) install.packages("BiocManager")
+#BiocManager::install("StructuralVariantAnnotation")
+#install.packages("stringr")
+library(VariantAnnotation)
+library(StructuralVariantAnnotation)
+library(stringr)
+
+args <- commandArgs()
+input_vcf <- args[1]
+output_vcf <- args[2]
+
+#' Simple SV type classifier
+simpleEventType <- function(gr) {
+    pgr = partner(gr)
+    return(ifelse(seqnames(gr) != seqnames(pgr), "CTX", # inter-chromosomosal
+        ifelse(strand(gr) == strand(pgr), "INV",
+        ifelse(gr$insLen >= abs(gr$svLen) * 0.7, "INS", # TODO: improve classification of complex events
+        ifelse(xor(start(gr) < start(pgr), strand(gr) == "-"), "DEL",
+        "DUP")))))
+}
+
+vcf <- readVcf(input_vcf, "hg38")
+info(header(vcf)) = unique(as(rbind(as.data.frame(info(header(vcf))), data.frame(
+    row.names=c("SIMPLE_TYPE"),
+    Number=c("1"),
+    Type=c("String"),
+    Description=c("Simple event type annotation based purely on breakend position and orientation."))), "DataFrame"))
+gr <- breakpointRanges(vcf)
+svtype <- simpleEventType(gr)
+info(vcf)$SIMPLE_TYPE <- NA_character_
+info(vcf[gr$sourceId])$SIMPLE_TYPE <- svtype
+info(vcf[gr$sourceId])$SVLEN <- gr$svLen
+writeVcf(vcf, output_vcf) # generated by example/gridss.sh
+
+# # TODO: perform event filtering here
+# # By default, GRIDSS is very sensitive but this comes at the cost of a high false discovery rate
+# gr <- gr[gr$FILTER == "PASS" & partner(gr)$FILTER == "PASS"] # Remove low confidence calls
+
+# simplegr <- gr[simpleEventType(gr) %in% c("INS", "INV", "DEL", "DUP")]
+# simplebed <- data.frame(
+#     chrom=seqnames(simplegr),
+#     # call the centre of the homology/inexact interval
+#     start=as.integer((start(simplegr) + end(simplegr)) / 2),
+#     end=as.integer((start(partner(simplegr)) + end(partner(simplegr))) / 2),
+#     name=simpleEventType(simplegr),
+#     score=simplegr$QUAL,
+#     strand="."
+# )
+# # Just the lower of the two breakends so we don't output everything twice
+# simplebed <- simplebed[simplebed$start < simplebed$end,]
+# write.table(simplebed, "chr12.1527326.DEL1024.simple.bed", quote=FALSE, sep='\t', row.names=FALSE, col.names=FALSE)

bin/viola_standardize.py

@@ -0,0 +1,37 @@
+#!/usr/local/bin/python
+
+import argparse
+import os
+
+import viola
+
+if __name__ == "__main__":
+    # Setting up argparser
+    parser = argparse.ArgumentParser(description="A script to standardize VCFs using Viola-SV")
+    parser.add_argument('vcf', metavar='FILE', type=str, help="The called VCF")
+    parser.add_argument('caller', metavar='STRING', type=str, help="The caller used to call the VCF")
+    parser.add_argument('out_file', metavar='FILE', type=str, help="The standardized VCF")
+    parser.add_argument('patient_name', metavar='STRING', type=str, help="The name of the patient in the VCF file")
+
+    args = parser.parse_args()
+
+    vcf = args.vcf
+    caller = args.caller
+    out_file = args.out_file
+    patient_name = args.patient_name
+
+    if caller == "smoove": caller = "lumpy"
+
+    if caller == "gridss":
+        svlen_not_added = True
+        old_vcf = f'old_{vcf}'
+        os.rename(vcf, old_vcf)
+        with open(old_vcf, 'r') as old:
+            with open(vcf, 'w') as new:
+                for line in old.readlines():
+                    if line.startswith("##INFO") and svlen_not_added:
+                        svlen_not_added = False
+                        new.write("##INFO=<ID=SVLEN,Number=1,Type=Integer,Description=\"The length of the structural variant.\">\n")
+                    new.write(line.replace("CIRPOS", "CIEND"))
+
+    viola.read_vcf(vcf, variant_caller=caller, patient_name=patient_name).breakend2breakpoint().to_vcf(out_file)