5964 Segmentation fault

[ranijames]

Dear All,
I am using RNASEQC tool binary version rnaseqc.v2.3.1.linux from GitHub Releases. When I try to run the tool on human samples. It is throwing the following error,

The output (if any) follows:

/cluster/shadow/.lsbatch/1554797826.1499627.shell: line 1:  5964 Segmentation fault 

And the Total memory I request to the run the above tool is cluster was: 80000.00 MB. Any help/suggestions would be much appreciated. I have successfully used the same tool on the mouse samples.

[agraubert]

Well that's not good. Could you give me some more information?

• What Linux distro are you running?
• Has this version of rnaseqc worked on this computer before?
• Could you try running when the flag -vv and posting the output up to and including the error?

Hopefully this will help identify the problem

[ranijames]

Thanks for your reply, the answers for the your questions

What Linux distro are you running?

I was trying to run the tool on cluster environment, which is LSF batch system. The Linux distro:

3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 GNU/Linux

LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.5.1804 (Core) 
Release:	7.5.1804
Codename:	Core

Has this version of rnaseqc worked on this computer before?

Yes, on Mouse samples with mouse reference

Could you try running when the flag -vv and posting the output up to and including the error?

Reading GTF Features...
Processing GTF Features...
Finished processing GTF in 47 seconds
Checking bam header...
Parsing bam...
Time elapsed: 11; Alignments processed: 3500000
Time elapsed: 21; Alignments processed: 7250000
Time elapsed: 31; Alignments processed: 10750000
Time elapsed: 41; Alignments processed: 14250000
Time elapsed: 51; Alignments processed: 17750000
Time elapsed: 61; Alignments processed: 21250000
Time elapsed: 71; Alignments processed: 24750000
Time elapsed: 81; Alignments processed: 28250000
Time elapsed: 91; Alignments processed: 31500000
Time elapsed: 101; Alignments processed: 34750000
Time elapsed: 111; Alignments processed: 38000000
Time elapsed: 121; Alignments processed: 41750000
Time elapsed: 131; Alignments processed: 45250000
Time elapsed: 141; Alignments processed: 48750000
Time elapsed: 151; Alignments processed: 52000000
Time elapsed: 161; Alignments processed: 55250000
Time elapsed: 171; Alignments processed: 59000000
Segmentation fault

[agraubert]

Unfortunately, I was unable to reproduce the crash, so I need some more information.

1. Could you show the command line arguments you gave to rnaseqc when it crashed?
2. If the program produced a core dump, would you mind uploading it?

[ranijames]

1. Could you show the command line arguments you gave to rnaseqc when it crashed?

rnaseqc.v2.3.1.linux -vv /annotation/gencode.v28lift37.annotation.gtf /BAM_files/EOC04_L1Aligned.sortedByCoord.out.bam /alignment/RNA-SeqC/EOC04_L1Aligned.sortedByCoord.out.bam

2. If the program produced a core dump, would you mind uploading it?

The program produced the above error. I am attaching all errors which I have. No, there was no core dump produced after the run.

[agraubert]

Okay. I still can't reproduce the crash. Would it be possible for you to provide a copy of the input gtf and bam?

If there are protected access concerns, it may be helpful to use a sed command to replace the read sequence with a garbage sequence of equal length. For example, with a read length of 76:

samtools view -h bam_file | sed 's/[ACTG]\{76\}/NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN/' | samtools view -hb > output_path.bam

[ranijames]

Thanks for the reply, sorry but we cannot share the data with third party. However, I have tried replacing the read sequence with garbage sequence, which also threw the following error,

*** Error in `/cluster/home/aalva/RNA_SeqC/rnaseqc.v2.3.1.linux': malloc(): memory corruption: 0x000000000091bd10 ***
Aborted

[agraubert]

That's fine. I kinda expected that would be the case. Without being able to touch the data, my only option is to ask you to debug this for me. If possible could you try the following:

1. Download and run Picard's ValidateSamFile to check for any corruption in the bam file
2. Download valgrind and run

valgrind --undef-value-errors=no rnaseqc.v2.3.1.linux -vv /annotation/gencode.v28lift37.annotation.gtf /BAM_files/EOC04_L1Aligned.sortedByCoord.out.bam /alignment/RNA-SeqC/EOC04_L1Aligned.sortedByCoord.out.bam

[agraubert]

Also, if you need to get rnaseqc to run in the interim, you could use the docker image gcr.io/broad-cga-aarong-gtex/rnaseqc:2.3.1

[ranijames]

Hello Aaron,
Sorry for the delay in reply. I have validated one of the BAM files using picard tools and i have few comments below,

## HISTOGRAM	java.lang.String
Error Type	Count
ERROR:HEADER_RECORD_MISSING_REQUIRED_TAG	1
ERROR:MATE_NOT_FOUND	489626
ERROR:MISSING_PLATFORM_VALUE	1

However, when I try with the first 1000 lines of BAm file it is not throwing the error. I can attach the gtf file used for the run. You can find the lines of gtf file where the execution is halting. https://drive.google.com/file/d/1OTox95OYdtqJ-pRSocePwN4r6j25wZUf/view?usp=sharing

[agraubert]

It looks like this is a full annotation, which is not recommended for RNA-SeQC. Reported counts may be inaccurate unless you're using a "collapsed" gene model, which you can generate by using the GTEX Annotation Collapsing Script

Looking at the ValidateSamFile output, there's nothing that jumps out as being an immediate culprit. It's odd that there are so many reads with missing mates, but I can't think of a reason this would crash RNA-SeQC. If you've had a chance to run valgrind, a copy of that stack trace would be profoundly helpful as it would allow me to identify what the program was trying to do when it crashed.

Also, if you're able to identify the exact read which is causing the crash it would be helpful if you can share that read's metadata (tags, position, cigar, flags). If you don't have time to identify the culprit read, or can't share the metadata, that's fine

[ranijames]

Hello Aaron,

Thanks for reply, and I have found the culprit reads leading to the fault. I am sharing the BAM file with those 45 reads here with you : https://drive.google.com/file/d/1-_N-A9CVCIUqs7KzOwYpidvdYyq-13G7/view?usp=sharing

In addition to that I have ran the valgrind tool on that BAM file using the following command, the gtf file I used here is the one I shared with you previously (https://drive.google.com/file/d/1OTox95OYdtqJ-pRSocePwN4r6j25wZUf/view?usp=sharing)

valgrind --undef-value-errors=no /cluster/home/user/RNA_SeqC/rnaseqc.v2.3.1.linux -vv /cluster/work/annotation/gencode.v28lift37.annotation.noChr.debug_fail.gtf /cluster/work/rnaseqc_debug_chr1_15966789-1596790.bam /cluster/work/tmp_user/Quality_assement/RNAseqC/rnaseqc_debug_chr1_15966789-1596790.bam

Which has thrown following error,

==2683==                                                                                                           
Reading GTF Features...                                                                                             
Processing GTF Features...                                                                                          
Finished processing GTF in 0 seconds                                                                                │
Checking bam header...                                                                                              │
Parsing bam...                                                                                                      │
Time Elapsed: 0; Alignments processed: 45                                                                           │
Total runtime: 0; Total CPU Time: 0                                                                                 │
Average Reads/Sec: inf                                                                                              │
Estimating library complexity...                                                                                    │
Generating report                                                                                                   │
*** Error in `/cluster/home/user/RNA_SeqC/rnaseqc.v2.3.1.linux': munmap_chunk(): invalid pointer: 0x00000000040523f│
0 ***                                                                                                               │
==2683==                                                                                                            │
==2683== Process terminating with default action of signal 6 (SIGABRT)                                              │
==2683==    at 0x61A567: raise (in /cluster/home/user/RNA_SeqC/rnaseqc.v2.3.1.linux)                               │
==2683==    by 0x5A3547: abort (in /cluster/home/user/RNA_SeqC/rnaseqc.v2.3.1.linux)                               │
==2683==    by 0x5B009F: __libc_message (in /cluster/home/user/RNA_SeqC/rnaseqc.v2.3.1.linux)                      │
==2683==    by 0x5B8D52: malloc_printerr (in /cluster/home/user/RNA_SeqC/rnaseqc.v2.3.1.linux)                     │
==2683==    by 0x430892: std::_Rb_tree<std::string, std::pair<std::string const, unsigned long>, std::_Select1st<std│
::pair<std::string const, unsigned long> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsi│
gned long> > >::_M_erase(std::_Rb_tree_node<std::pair<std::string const, unsigned long> >*) (in /cluster/home/aalva/│
RNA_SeqC/rnaseqc.v2.3.1.linux)                                                                                      │
==2683==    by 0x40A3CC: main (in /cluster/home/user/RNA_SeqC/rnaseqc.v2.3.1.linux)                                │
==2683==                                                                                                            │
==2683== HEAP SUMMARY:                                                                                              │
==2683==     in use at exit: 0 bytes in 0 blocks                                                                    │
==2683==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated                                                   │
==2683==                                                                                                            │
==2683== All heap blocks were freed -- no leaks are possible                                                        │
==2683==                                                                                                            │
==2683== For counts of detected and suppressed errors, rerun with: -v                                               │
==2683== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)                                             │
Aborted                                                    

Please let me know if you need any further information.

[agraubert]

Thank you so much for this information! I was able to reproduce the crash on my CentOS 7 VM. I'll let you know as soon as I have any updates

[agraubert]

I believe I have identified the problem.

It appears that there are two exons with the same id:

1       HAVANA  exon    1595868 1597458 .       -       .       gene_id "ENSG00000189339.11_2"; transcript_id "ENST00000617444.4_1"; gene_type "protein_coding"; gene_name "SLC35E2B"; transcript_type "protein_coding"; transcript_name "RP11-345P4.4-003"; exon_number 10; exon_id "ENSE00003730248.1_1"; level 2; protein_id "ENSP00000481694.1"; transcript_support_level 1; tag "alternative_5_UTR"; tag "basic"; tag "appris_principal_1"; tag "CCDS"; ccdsid "CCDS44041.1"; havana_gene "OTTHUMG00000078639.3_2"; havana_transcript "OTTHUMT00000171589.3_1"; remap_original_location "chr1:-:1664429-1666019"; remap_status "partial";
1       HAVANA  exon    1594842 1595855 .       -       .       gene_id "ENSG00000189339.11_2"; transcript_id "ENST00000617444.4_1"; gene_type "protein_coding"; gene_name "SLC35E2B"; transcript_type "protein_coding"; transcript_name "RP11-345P4.4-003"; exon_number 11; exon_id "ENSE00003730248.1_1"; level 2; protein_id "ENSP00000481694.1"; transcript_support_level 1; tag "alternative_5_UTR"; tag "basic"; tag "appris_principal_1"; tag "CCDS"; ccdsid "CCDS44041.1"; havana_gene "OTTHUMG00000078639.3_2"; havana_transcript "OTTHUMT00000171589.3_1"; remap_original_location "chr1:-:1663403-1664416"; remap_status "partial";

The second exon overwrites the length stored for the first one, which is causing problems later on. RNA-SeQC relies on the lengths when allocating memory for certain data structures. This causes memory to be written out of bounds (because the memory allocated for the first exon is ~500 bytes shorter than it should be) which is causing the problem.

The reason that the exact circumstances of the crash vary from environment to environment is that the state and layout of the program memory is highly dependent on the environment, which causes this problem to manifest in different ways (or not at all) on different platforms.

I am going to release a patch to RNA-SeQC which should avoid both the memory issue and print an error if duplicate gene/exon ids are detected

[agraubert]

The fixes have been incorporated into RNA-SeQC 2.3.2

If this does not fix your issue, please feel free to comment and reopen this issue

[ranijames]

Hello Aaron,

I have issues installing from the new release version of the tool. I
downloaded the tool, and then unzipped and tried

cd Seqlib and then tried following

configure

  make

make install

bash: ./rnaseqc: No such file or directory

Then when I check the installation it says command not found. Do you have
any binary version released for the same?

Thanks

[agraubert]

If you're building from source, you don't need to cd into the seqlib directory. The rnaseqc makefile is supposed to build seqlib for you. That said, the Linux static binary is available on the releases page

I apologize if it wasn't there when you first looked; it can take ~30 minutes after a new release is published before the automated build system attaches the binaries

[ranijames]

Hello Aaron,
Thank you so much, for your reply. I have now got the new release tool in my machine and tried it on my samples using the same gtf file. The following is the command I tried,

# LSBATCH: User input
/cluster/home/user/rnaseqc.v2.3.2.linux  /gencode.v28lift37.annotation.gtf /RCC16_L1Aligned.sortedByCoord.out.bam /RCC16_L1

The above run, however, is throwing an error for the duplicated exon,

Failed to parse the GTF: Detected non-unique Exon ID: ENSE00001957285.1_1

[ranijames]

Hello Aaron, Thank you so much, for your reply. I have now got the new release tool in my machine and tried it on my samples using the same gtf file. The following is the command I tried. # LSBATCH: User input /cluster/home/user/rnaseqc.v2.3.2.linux /gencode.v28lift37.annotation.gtf /RCC16_L1Aligned.sortedByCoord.out.bam /RCC16_L1 The above run, however, is throwing an error for the duplicated exon, Failed to parse the GTF: Detected non-unique Exon ID: ENSE00001957285.1_1

…

On Thu, Apr 18, 2019 at 2:04 PM Aaron Graubert ***@***.***> wrote: If you're building from source, you don't need to cd into the seqlib directory. The rnaseqc makefile is supposed to build seqlib for you. That said, the Linux static binary is available on the releases page <https://github.com/broadinstitute/rnaseqc/releases/tag/v2.3.2> I apologize if it wasn't there when you first looked; it can take ~30 minutes after a new release is published before the automated build system attaches the binaries — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#26 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB4I6JLFW755MCKOFOUKFHTPRBPTXANCNFSM4HEQBD3Q> .

-- Kind Regards, Dr.Alva Rani James

[agraubert]

That is correct.

Ensembl IDs are always unique, and RNA-SeQC relies on this fact. Your GTF contains two exons (shown above), with the same exon ID. You must change them to have correct IDs.

Additionally, you're using a full gencode annotation, which will yield incorrect results. You should use the GTEx Collapse Annotation Script to convert your full annotation to a collapsed model which will work for RNA-SeQC

[ranijames]

Hello All, Is it possible to install RNAseqC via conda environment?

…

On Thu, Apr 18, 2019 at 6:53 PM Aaron Graubert ***@***.***> wrote: That is correct. Ensembl IDs are always unique, and RNA-SeQC relies on this fact. Your GTF contains two exons (shown above), with the same exon ID. You must change them to have correct IDs. Additionally, you're using a full gencode annotation, which will yield incorrect results. You should use the GTEx Collapse Annotation Script <https://github.com/broadinstitute/gtex-pipeline/tree/master/gene_model> to convert your full annotation to a collapsed model which will work for RNA-SeQC — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#26 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB4I6JIVAZPVBWGMWM54D5DPRCRQLANCNFSM4HEQBD3Q> .

-- Kind Regards, Dr.Alva Rani James

[agraubert]

No, but at some point it may become available (#19). In the meantime, you can download the static binaries, use the docker image gcr.io/broad-cga-aarong-gtex/rnaseqc:latest, or build from source

[ranijames]

I am currently intergrating soem tools and buliding a SNAKEMAKE pipeline. I have a lits of tools and packages i mentioned in my enviorment file. So if RNA-seqC can be downloaded via conda or pip or via any dependencies I could mention that in my enviroment.yml file.

…

On Mon, Jul 8, 2019 at 4:03 PM Aaron Graubert ***@***.***> wrote: No, but at some point it may become available (#19 <#19>). In the meantime, you can download the static binaries <https://github.com/broadinstitute/rnaseqc/releases>, use the docker image gcr.io/broad-cga-aarong-gtex/rnaseqc:latest, or build from source — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#26>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB4I6JJ55N45RP4KABPDYKLP6NCLLANCNFSM4HEQBD3Q> .

-- Kind Regards, Dr.Alva Rani James