throw new Exception("Cannot append SCENIC loom to SCope loom because the IDs do not match.")[BUG]

[LinearParadox]

Describe the bugDescribe the bug*
Program crashes when trying to merge SCOPE and filtered looms.

To Reproduce
Steps to reproduce the behavior:

1. Configure with these options:
   nextflow -C nextflow.config run vib-singlecell-nf/vsn-pipelines -entry bbknn_scenic

Additionally, the file containing the filtered loom is deleted and has to be recovered.

• OS: [Ubuntu]

This seems to be an issue with h5ad+to_loom.py, as the file it creates is not compliant with loom standards. SPecifically, opening it in loompy gives the following:
ValueError: Row attribute 'ClusterMarkers_0' dtype [('0', '<i8'), ('1', '<i8'), ('2', '<i8'), ('3', '<i8'), ('4', '<i8'), ('5', '<i8'), ('6', '<i8'), ('7', '<i8'), ('8', '<i8'), ('9', '<i8'), ('10', '<i8'), ('11', '<i8'), ('12', '<i8')] is not allowed
Row attribute 'ClusterMarkers_0_avg_logFC' dtype [('0', '<f8'), ('1', '<f8'), ('2', '<f8'), ('3', '<f8'), ('4', '<f8'), ('5', '<f8'), ('6', '<f8'), ('7', '<f8'), ('8', '<f8'), ('9', '<f8'), ('10', '<f8'), ('11', '<f8'), ('12', '<f8')] is not allowed
Row attribute 'ClusterMarkers_0_pval' dtype [('0', '<f8'), ('1', '<f8'), ('2', '<f8'), ('3', '<f8'), ('4', '<f8'), ('5', '<f8'), ('6', '<f8'), ('7', '<f8'), ('8', '<f8'), ('9', '<f8'), ('10', '<f8'), ('11', '<f8'), ('12', '<f8')] is not allowed
Column attribute 'Clusterings' dtype [('0', '<i8')] is not allowed
Column attribute 'Embedding' dtype [('_X', '<f4'), ('_Y', '<f4')] is not allowed
Column attribute 'Embeddings_X' dtype [('-1', '<f4'), ('0', '<f8'), ('1', '<f4')] is not allowed
Column attribute 'Embeddings_Y' dtype [('-1', '<f4'), ('0', '<f8'), ('1', '<f4')] is not allowed

[cchen22]

I am facing the same issue. Did you solve it @LinearParadox?

Thanks,
Chen

[KrisDavie]

Hey @LinearParadox, @cchen22 ,

Sorry for the delayed response. The exception you are seeing is something that shows up when almost any error happens, I'd need to see a full error log to see what might have gone wrong.

You are correct that we use a non-compliant format for the looms, the validation was added to loompy after we had already decided on a file format. You can still load the loom files by passing validate=False to the lp.connect function.

Cheers,

Kris

[ccasar]

Not sure if this is the same issue, but I'm also running into the same error message with my own data as well as when following this tutorial when running the single_sample_scenic entry.

I've tried with both the suggested vsn version 0.25.0 and the newest version.

Without deeper knowledge of the workflow it seems that something is wrong with the dependencies between the nextflow process since the workflow breaks here:

vsn-pipelines/src/scenic/main.nf

Line 194 in 96cff8e

throw new Exception("Cannot append SCENIC loom to SCope loom because the IDs do not match.")

while there are still arboreto process running in the background. Waiting for these to finish and then resuming the pipeline also leads to the same error.

[jflucier]

I have same error as @ccasar when running tuotiral pipeline https://vsn-pipelines-examples.readthedocs.io/en/latest/PBMC10k.html

As this been solved?

I have attached nextflow log. If other log require, please ask.

nextflow.log

thanks

[ccasar]

As @KrisDavie mentioned the given error message is pretty generic.
In my case it turned that the recommended mc_v10_clust files from the resources website were not compatible with the current pipeline version.

To fix these problems I switched to a more recent scenic container version in my nextflow config file, e.g.

scenic {
         container = 'aertslab/pyscenic_scanpy:0.12.1_1.9.1'
         skipReports = true
}

The SCENIC report step is not compatible with this container version, so I had to skip it.

Additionally some smaller changes needed to be made when importing packages during the scenic steps. You can find these changes in my vsn-pipeline fork at https://github.com/ccasar/vsn-pipelines

[jflucier]

thanks very much @ccasar

got it working using your vsn-pipeline fork.