Help with external counts in sample sheet

[phillip-richmond-alamya]

Hello,

I've fought with this for a few days now and can't figure out how to get the external samples to play nicely. When I run with BAM files it works fine, but I can't get the samples in the external dataset to be incorporated.

Here's my latest error:

WARNING: 13 files missing in samples annotation. Ignoring...
check for missing R packages
MonoallelicExpression has been turned off in the config file
rnaVariantCalling has been turned off in the config file
Structuring dependencies...
Dependencies file generated at: /mnt/scratch/Public/WorkLearn2022/RunDropOnSamplesNoGeuvadis/tmp/tmpb1_1d8oz

Building DAG of jobs...
WorkflowError:
WorkflowError:
    WorkflowError (rule AberrantExpression_pipeline_Counting_mergeCounts_R, line 135, /mnt/scratch/Public/WorkLearn2022/RunDropOnSamplesNoGeuvadis/tmp/tmpb1_1d8oz):
        Function did not return str or list of str.
    MissingInputException: Missing input files for rule markdown:
        output: /mnt/scratch/Public/WorkLearn2022/RunDropOnSamplesNoGeuvadis/Reports/AberrantExpression/Counting/Gencode29/Summary_outrider.html
        wildcards: file=AberrantExpression/Counting/Gencode29/Summary_outrider
        affected files:
            AberrantExpression/Counting/Gencode29/Summary_outrider.md
MissingInputException: Missing input files for rule markdown:
    output: /mnt/scratch/Public/WorkLearn2022/RunDropOnSamplesNoGeuvadis/Reports/Scripts_AberrantExpression_pipeline_Counting_Datasets.html
    wildcards: file=Scripts_AberrantExpression_pipeline_Counting_Datasets
    affected files:
        Scripts_AberrantExpression_pipeline_Counting_Datasets.md

Here's my config:

projectTitle: "AS Analysis of CHR Positive Control Samples Aligned with STAR"
root: "/mnt/scratch/Public/Worklearn2022/RunDropOnSamplesNoGeuvadis/"
htmlOutputPath: "/mnt/scratch/Public/WorkLearn2022/RunDropOnSamplesNoGeuvadis/Reports"
indexWithFolderName: true 

root: output
sampleAnnotation: "/mnt/scratch/Public/WorkLearn2022/RunDropOnSamplesNoGeuvadis/metadata/sample_annotation_table_gtex_and_trios_2022-01-23.tsv"
geneAnnotation:
    Gencode29: "/mnt/scratch/Public/WorkLearn2022/DropReferences/gencode.v38.annotation.gtf"
genomeAssembly: hg38
genome: "/mnt/scratch/Public/WorkLearn2022/DropReferences/GRCh38.primary_assembly.genome.fa"

hpoFile: null

exportCounts:
    # specify which gene annotations to include and which
    # groups to exclude when exporting counts
    geneAnnotations:
        - Gencode29

aberrantExpression:
    run: true
    groups:
        - outrider
    fpkmCutoff: 1
    implementation: autoencoder
    padjCutoff: 1
    zScoreCutoff: 0
    maxTestedDimensionProportion: 3
    dassie:
        tssWindow: 500
        pasWindow: 1000

aberrantSplicing:
    run: true
    groups:
        - fraser
    recount: true
    longRead: false
    keepNonStandardChrs: false
    filter: true
    minExpressionInOneSample: 20
    minDeltaPsi: 0.05
    implementation: PCA
    padjCutoff: 1
    zScoreCutoff: 0
    deltaPsiCutoff : 0.05
    maxTestedDimensionProportion: 6

mae:
    run: false
    groups:
        - mae
    gatkIgnoreHeaderCheck: true
    padjCutoff: .05
    allelicRatioCutoff: 0.8
    addAF: false
    maxAF: .001
    maxVarFreqCohort: .04
    # VCF-BAM matching
    qcVcf: /mnt/scratch/Public/WorkLearn2022/DropReferences/qc_vcf_1000G_hg38.vcf.gz
    qcGroups:
      - mae

rnaVariantCalling:
    run: false
    groups:
        - varcall
    highQualityVCFs:
        - /mnt/scratch/Public/WorkLearn2022/DropReferences/Mills_and_1000G_gold_standard.indels.hg38.sites.chrPrefix.vcf.gz
        - /mnt/scratch/Public/WorkLearn2022/DropReferences/1000G_phase1.snps.high_confidence.hg38.sites.chrPrefix.vcf.gz
    dbSNP: Data/00-All.vcf.gz
    repeat_mask: /mnt/scratch/Public/WorkLearn2022/DropReferences/hg38_repeatMasker_sorted.chrPrefix.bed
    createSingleVCF: true                    
    addAF: true
    maxAF: .001
    maxVarFreqCohort: .1
    hcArgs: 
    minAlt: 3

tools:
    gatkCmd: gatk
    bcftoolsCmd: bcftools
    samtoolsCmd: samtools

Attached is my sample sheet (also first 50 lines pasted here)
SampleAnnotationFile.txt

RNA_ID	DNA_ID	DROP_GROUP	RNA_BAM_FILE	GENE_COUNTS_FILE	GENE_ANNOTATION	SPLICE_COUNTS_DIR	DNA_VCF_FILE	PAIRED_END	COUNT_MODE	COUNT_OVERLAPS	STRAND
DH141401		fraser,outrider,samples_bcchr	/mnt/scratch/Public/WorkLearn2022/Data/samples_bcchr_star/DH141401_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
DH141402		fraser,outrider,samples_bcchr	/mnt/scratch/Public/WorkLearn2022/Data/samples_bcchr_star/DH141402_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
DH141403		fraser,outrider,samples_bcchr	/mnt/scratch/Public/WorkLearn2022/Data/samples_bcchr_star/DH141403_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
DH1414HC		fraser,outrider,samples_bcchr	/mnt/scratch/Public/WorkLearn2022/Data/samples_bcchr_star/DH1414HC_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
DH242401		fraser,outrider,samples_bcchr	/mnt/scratch/Public/WorkLearn2022/Data/samples_bcchr_star/DH242401_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
DH242402		fraser,outrider,samples_bcchr	/mnt/scratch/Public/WorkLearn2022/Data/samples_bcchr_star/DH242402_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
DH242403		fraser,outrider,samples_bcchr	/mnt/scratch/Public/WorkLearn2022/Data/samples_bcchr_star/DH242403_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
DH313201		fraser,outrider,samples_bcchr	/mnt/scratch/Public/WorkLearn2022/Data/samples_bcchr_star/DH313201_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
DH313202		fraser,outrider,samples_bcchr	/mnt/scratch/Public/WorkLearn2022/Data/samples_bcchr_star/DH313202_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
DH313203		fraser,outrider,samples_bcchr	/mnt/scratch/Public/WorkLearn2022/Data/samples_bcchr_star/DH313203_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
DH495101		fraser,outrider,samples_bcchr	/mnt/scratch/Public/WorkLearn2022/Data/samples_bcchr_star/DH495101_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
DH495102		fraser,outrider,samples_bcchr	/mnt/scratch/Public/WorkLearn2022/Data/samples_bcchr_star/DH495102_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
DH495103		fraser,outrider,samples_bcchr	/mnt/scratch/Public/WorkLearn2022/Data/samples_bcchr_star/DH495103_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
HG00099		fraser,controls_1kG	/mnt/scratch/Public/WorkLearn2022/align-rna/OUTPUT_DIR/HG00099_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
HG00106		fraser,controls_1kG	/mnt/scratch/Public/WorkLearn2022/align-rna/OUTPUT_DIR/HG00106_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
HG00110		fraser,controls_1kG	/mnt/scratch/Public/WorkLearn2022/align-rna/OUTPUT_DIR/HG00110_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
HG00115		fraser,controls_1kG	/mnt/scratch/Public/WorkLearn2022/align-rna/OUTPUT_DIR/HG00115_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
HG00117		fraser,controls_1kG	/mnt/scratch/Public/WorkLearn2022/align-rna/OUTPUT_DIR/HG00117_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
HG00126		fraser,controls_1kG	/mnt/scratch/Public/WorkLearn2022/align-rna/OUTPUT_DIR/HG00126_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
HG00132		fraser,controls_1kG	/mnt/scratch/Public/WorkLearn2022/align-rna/OUTPUT_DIR/HG00132_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
HG00136		fraser,controls_1kG	/mnt/scratch/Public/WorkLearn2022/align-rna/OUTPUT_DIR/HG00136_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
HG00149		fraser,controls_1kG	/mnt/scratch/Public/WorkLearn2022/align-rna/OUTPUT_DIR/HG00149_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
HG00151		fraser,controls_1kG	/mnt/scratch/Public/WorkLearn2022/align-rna/OUTPUT_DIR/HG00151_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
HG00178		fraser,controls_1kG	/mnt/scratch/Public/WorkLearn2022/align-rna/OUTPUT_DIR/HG00178_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
HG00179		fraser,controls_1kG	/mnt/scratch/Public/WorkLearn2022/align-rna/OUTPUT_DIR/HG00179_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
HG00187		fraser,controls_1kG	/mnt/scratch/Public/WorkLearn2022/align-rna/OUTPUT_DIR/HG00187_dup_marked.bam		Gencode29			TRUE	IntersectionStrict	TRUE	no
GTEX-111YS-0006-SM-5NQBE		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-1122O-0005-SM-5O99J		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-1128S-0005-SM-5P9HI		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-113IC-0006-SM-5NQ9C		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-113JC-0006-SM-5O997		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-117XS-0005-SM-5PNU6		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-117YW-0005-SM-5NQ8Z		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-1192W-0005-SM-5NQBQ		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-1192X-0005-SM-5NQC3		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-11DXW-0006-SM-5NQ7Y		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-11DXX-0005-SM-5NQ8B		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-11DXY-0006-SM-5NQ8N		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-11DXZ-0006-SM-5LZZH		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-11DYG-0006-SM-5NQB2		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-11EI6-0005-SM-5N9GN		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-11EM3-0005-SM-5N9DK		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-11EMC-0006-SM-5O9DN		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-11EQ8-0006-SM-5P9HJ		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-11EQ9-0006-SM-5LUB6		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-11GS4-0006-SM-5SI9M		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-11GSO-0005-SM-5O9CA		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-11GSP-0006-SM-5N9EL		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/					
GTEX-11I78-0005-SM-5N9GB		fraser,outrider,controls_gtex_blood		/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/geneCounts.tsv.gz	Gencode29	/mnt/scratch/Public/WorkLearn2022/Data/controls_gtex/exported_counts/Whole_Blood--GRCh38--gencode29/	

Thanks for you help and this awesome tool!

[vyepez88]

Hi @phillip-richmond-alakai , sorry for not coming back to you before. We weren't able to reproduce this error until last week. It's super weird as the aberrant splicing pipeline does work. We're investigating it and try to come back to you soon.

[Phillip-a-richmond]

If there's a config and sample sheet you've gotten to work where you have the external matrices as controls then I can try that as well?

Thanks for the support!

-Phil

[vyepez88]

Hi, yes, when you execute drop demo, you get an example config and sample sheet. That works. It also works in most datasets. Can you try whether the aberrant splicing module works for you?

[vyepez88]

Hi @Phillip-a-richmond, I think I know what the error is, can you try removing the value Gencode29 from the GENE_ANNOTATION column for the samples that do have a BAM file?