(scenicplus) [gilgolan@localhost Snakemake]$ snakemake --cores 20 Assuming unrestricted shared filesystem usage for local execution. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 20 Rules claiming more threads will be scaled down. Job stats: job count --------------------------- ------- AUCell_direct 1 AUCell_extended 1 all 1 download_genome_annotations 1 eGRN_direct 1 eGRN_extended 1 get_search_space 1 motif_enrichment_cistarget 1 motif_enrichment_dem 1 prepare_GEX_ACC_multiome 1 prepare_menr 1 region_to_gene 1 scplus_mudata 1 tf_to_gene 1 total 14 Select jobs to execute... Execute 2 jobs... [Mon Oct 14 16:09:31 2024] localrule download_genome_annotations: output: genome_annotation.tsv, chromsizes.tsv jobid: 8 reason: Missing output files: chromsizes.tsv, genome_annotation.tsv resources: tmpdir=/tmp [Mon Oct 14 16:09:31 2024] localrule prepare_GEX_ACC_multiome: input: /home/gilgolan/bioinfo_analysis/scenicplus/tutorial2/outs/cistopic_obj.pkl, /home/gilgolan/bioinfo_analysis/scenicplus/tutorial2/scRNAseq/adata.h5ad output: ACC_GEX.h5mu jobid: 2 reason: Missing output files: ACC_GEX.h5mu resources: tmpdir=/tmp 2024-10-14 16:09:33,897 SCENIC+ INFO Reading cisTopic object. 2024-10-14 16:09:34,253 SCENIC+ INFO Reading gene expression AnnData. 2024-10-14 16:09:34,336 Ingesting multiome data INFO Found 1963 multiome cells. 2024-10-14 16:09:34,462 cisTopic INFO Imputing region accessibility 2024-10-14 16:09:34,463 cisTopic INFO Impute region accessibility for regions 0-20000 2024-10-14 16:09:34,645 cisTopic INFO Impute region accessibility for regions 20000-40000 2024-10-14 16:09:34,845 cisTopic INFO Impute region accessibility for regions 40000-60000 2024-10-14 16:09:35,028 cisTopic INFO Impute region accessibility for regions 60000-80000 2024-10-14 16:09:35,216 cisTopic INFO Impute region accessibility for regions 80000-100000 2024-10-14 16:09:35,418 cisTopic INFO Impute region accessibility for regions 100000-120000 2024-10-14 16:09:35,619 cisTopic INFO Impute region accessibility for regions 120000-140000 2024-10-14 16:09:35,806 cisTopic INFO Impute region accessibility for regions 140000-160000 2024-10-14 16:09:35,993 cisTopic INFO Impute region accessibility for regions 160000-180000 2024-10-14 16:09:36,194 cisTopic INFO Impute region accessibility for regions 180000-200000 2024-10-14 16:09:36,389 cisTopic INFO Impute region accessibility for regions 200000-220000 2024-10-14 16:09:36,585 cisTopic INFO Impute region accessibility for regions 220000-240000 2024-10-14 16:09:36,788 cisTopic INFO Impute region accessibility for regions 240000-260000 2024-10-14 16:09:36,982 cisTopic INFO Impute region accessibility for regions 260000-280000 2024-10-14 16:09:37,182 cisTopic INFO Impute region accessibility for regions 280000-300000 2024-10-14 16:09:37,374 cisTopic INFO Impute region accessibility for regions 300000-320000 2024-10-14 16:09:37,558 cisTopic INFO Impute region accessibility for regions 320000-340000 2024-10-14 16:09:37,773 cisTopic INFO Impute region accessibility for regions 340000-360000 2024-10-14 16:09:37,971 cisTopic INFO Impute region accessibility for regions 360000-380000 2024-10-14 16:09:38,160 cisTopic INFO Impute region accessibility for regions 380000-400000 2024-10-14 16:09:38,346 cisTopic INFO Impute region accessibility for regions 400000-420000 2024-10-14 16:09:38,531 cisTopic INFO Impute region accessibility for regions 420000-440000 2024-10-14 16:09:38,691 cisTopic INFO Done! ... storing 'sample_id' as categorical ... storing 'VSN_cell_type' as categorical ... storing 'VSN_leiden_res0.3' as categorical ... storing 'VSN_leiden_res0.6' as categorical ... storing 'VSN_leiden_res0.9' as categorical ... storing 'VSN_leiden_res1.2' as categorical ... storing 'VSN_sample_id' as categorical ... storing 'Seurat_leiden_res0.6' as categorical ... storing 'Seurat_leiden_res1.2' as categorical ... storing 'Seurat_cell_type' as categorical ... storing 'Chromosome' as categorical [Mon Oct 14 16:09:45 2024] Finished job 2. 1 of 14 steps (7%) done Select jobs to execute... 2024-10-14 16:09:54,769 Download gene annotation INFO Using genome: GRCh38.p14 2024-10-14 16:09:54,772 Download gene annotation INFO Found corresponding genome Id 51 on NCBI 2024-10-14 16:09:55,277 Download gene annotation INFO Found corresponding assembly Id 11968211 on NCBI 2024-10-14 16:09:55,790 Download gene annotation INFO Downloading assembly information from: http://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/001/405/GCF_000001405.40_GRCh38.p14/GCF_000001405.40_GRCh38.p14_assembly_report.txt 2024-10-14 16:09:57,102 Download gene annotation INFO Found following assembled molecules (chromosomes): 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 X Y MT 2024-10-14 16:09:57,112 Download gene annotation INFO Converting chromosomes names to UCSC style as follows: Original UCSC 1 chr1 2 chr2 3 chr3 4 chr4 5 chr5 6 chr6 7 chr7 8 chr8 9 chr9 10 chr10 11 chr11 12 chr12 13 chr13 14 chr14 15 chr15 16 chr16 17 chr17 18 chr18 19 chr19 20 chr20 21 chr21 22 chr22 X chrX Y chrY MT chrM 2024-10-14 16:09:57,127 SCENIC+ INFO Saving chromosome sizes to: chromsizes.tsv 2024-10-14 16:09:57,128 SCENIC+ INFO Saving genome annotation to: genome_annotation.tsv [Mon Oct 14 16:09:57 2024] Finished job 8. 2 of 14 steps (14%) done Execute 1 jobs... [Mon Oct 14 16:09:57 2024] localrule get_search_space: input: ACC_GEX.h5mu, genome_annotation.tsv, chromsizes.tsv output: search_space.tsv jobid: 11 reason: Missing output files: search_space.tsv; Input files updated by another job: chromsizes.tsv, genome_annotation.tsv, ACC_GEX.h5mu resources: tmpdir=/tmp 2024-10-14 16:09:59,876 SCENIC+ INFO Reading data /home/gilgolan/.local/lib/python3.11/site-packages/anndata/_core/anndata.py:522: FutureWarning: The dtype argument is deprecated and will be removed in late 2024. warnings.warn( /home/gilgolan/.local/lib/python3.11/site-packages/anndata/_core/anndata.py:522: FutureWarning: The dtype argument is deprecated and will be removed in late 2024. warnings.warn( 2024-10-14 16:10:01,902 Get search space INFO Extending promoter annotation to 10 bp upstream and 10 downstream 2024-10-14 16:10:02,005 Get search space INFO Extending search space to: 150000 bp downstream of the end of the gene. 150000 bp upstream of the start of the gene. 2024-10-14 16:10:02,440 Get search space INFO Intersecting with regions. 2024-10-14 16:10:03,751 Get search space INFO Calculating distances from region to gene 2024-10-14 16:10:45,521 Get search space INFO Imploding multiple entries per region and gene 2024-10-14 16:12:24,738 SCENIC+ INFO Writing search space to: search_space.tsv [Mon Oct 14 16:12:26 2024] Finished job 11. 3 of 14 steps (21%) done Select jobs to execute... Execute 1 jobs... [Mon Oct 14 16:12:26 2024] localrule region_to_gene: input: ACC_GEX.h5mu, search_space.tsv output: region_to_gene_adj.tsv jobid: 10 reason: Missing output files: region_to_gene_adj.tsv; Input files updated by another job: search_space.tsv, ACC_GEX.h5mu threads: 20 resources: tmpdir=/tmp 2024-10-14 16:12:31,545 SCENIC+ INFO Reading multiome MuData. /home/gilgolan/.local/lib/python3.11/site-packages/anndata/_core/anndata.py:522: FutureWarning: The dtype argument is deprecated and will be removed in late 2024. warnings.warn( /home/gilgolan/.local/lib/python3.11/site-packages/anndata/_core/anndata.py:522: FutureWarning: The dtype argument is deprecated and will be removed in late 2024. warnings.warn( 2024-10-14 16:12:33,092 SCENIC+ INFO Reading search space 2024-10-14 16:12:33,683 R2G INFO Calculating region to gene importances, using GBM method Running using 20 cores: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 18565/18565 [19:56<00:00, 15.52it/s] 2024-10-14 16:32:36,714 R2G INFO Calculating region to gene correlation, using SR method Running using 20 cores: 0%| | 0/18565 [00:00 return [func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/gilgolan/.local/lib/python3.11/site-packages/scenicplus/cli/commands.py", line 320, in _run_dem_single_region_set dem_db = DEMDatabase( ^^^^^^^^^^^^ File "/home/gilgolan/.local/lib/python3.11/site-packages/pycistarget/motif_enrichment_dem.py", line 147, in __init__ self.db_regions = pr.PyRanges(region_names_to_coordinates(list(self.genes))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gilgolan/.local/lib/python3.11/site-packages/pycistarget/utils.py", line 35, in region_names_to_coordinates regiondf.columns=['Chromosome', 'Start', 'End'] ^^^^^^^^^^^^^^^^ File "/home/gilgolan/.local/lib/python3.11/site-packages/pandas/core/generic.py", line 5920, in __setattr__ return object.__setattr__(self, name, value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "pandas/_libs/properties.pyx", line 69, in pandas._libs.properties.AxisProperty.__set__ File "/home/gilgolan/.local/lib/python3.11/site-packages/pandas/core/generic.py", line 822, in _set_axis self._mgr.set_axis(axis, labels) File "/home/gilgolan/.local/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 228, in set_axis self._validate_set_axis(axis, new_labels) File "/home/gilgolan/.local/lib/python3.11/site-packages/pandas/core/internals/base.py", line 70, in _validate_set_axis raise ValueError( ValueError: Length mismatch: Expected axis has 0 elements, new values have 3 elements """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/gilgolan/.local/bin/scenicplus", line 8, in sys.exit(main()) ^^^^^^ File "/home/gilgolan/.local/lib/python3.11/site-packages/scenicplus/cli/scenicplus.py", line 1137, in main args.func(args) File "/home/gilgolan/.local/lib/python3.11/site-packages/scenicplus/cli/scenicplus.py", line 588, in motif_enrichment_dem run_motif_enrichment_dem( File "/home/gilgolan/.local/lib/python3.11/site-packages/scenicplus/cli/commands.py", line 466, in run_motif_enrichment_dem dem_results: List[DEM] = joblib.Parallel( ^^^^^^^^^^^^^^^^ File "/home/gilgolan/.local/lib/python3.11/site-packages/joblib/parallel.py", line 1952, in __call__ return output if self.return_generator else list(output) ^^^^^^^^^^^^ File "/home/gilgolan/.local/lib/python3.11/site-packages/joblib/parallel.py", line 1595, in _get_outputs yield from self._retrieve() File "/home/gilgolan/.local/lib/python3.11/site-packages/joblib/parallel.py", line 1699, in _retrieve self._raise_error_fast() File "/home/gilgolan/.local/lib/python3.11/site-packages/joblib/parallel.py", line 1734, in _raise_error_fast error_job.get_result(self.timeout) File "/home/gilgolan/.local/lib/python3.11/site-packages/joblib/parallel.py", line 736, in get_result return self._return_or_raise() ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gilgolan/.local/lib/python3.11/site-packages/joblib/parallel.py", line 754, in _return_or_raise raise self._result ValueError: Length mismatch: Expected axis has 0 elements, new values have 3 elements [Mon Oct 14 16:36:07 2024] Error in rule motif_enrichment_dem: jobid: 7 input: /home/gilgolan/bioinfo_analysis/scenicplus/tutorial2/outs/region_sets, /home/gilgolan/bioinfo_analysis/scenicplus/tutorial2/cistarget_db/hg38_10kbp_up_10kbp_down_full_tx_v10_clust.genes_vs_motifs.scores.feather, genome_annotation.tsv, /home/gilgolan/bioinfo_analysis/scenicplus/tutorial2/motifs-v10nr_clust-nr.hgnc-m0.001-o0.0.tbl output: dem_results.hdf5, dem_results.html shell: scenicplus grn_inference motif_enrichment_dem --region_set_folder /home/gilgolan/bioinfo_analysis/scenicplus/tutorial2/outs/region_sets --dem_db_fname /home/gilgolan/bioinfo_analysis/scenicplus/tutorial2/cistarget_db/hg38_10kbp_up_10kbp_down_full_tx_v10_clust.genes_vs_motifs.scores.feather --output_fname_dem_result dem_results.hdf5 --temp_dir /tmp --species homo_sapiens --fraction_overlap_w_dem_database 0.4 --max_bg_regions 500 --genome_annotation genome_annotation.tsv --balance_number_of_promoters --promoter_space 1000 --adjpval_thr 0.05 --log2fc_thr 1.0 --mean_fg_thr 0.0 --motif_hit_thr 3.0 --path_to_motif_annotations /home/gilgolan/bioinfo_analysis/scenicplus/tutorial2/motifs-v10nr_clust-nr.hgnc-m0.001-o0.0.tbl --annotation_version v10nr_clust --motif_similarity_fdr 0.001 --orthologous_identity_threshold 0.0 --annotations_to_use Direct_annot Orthology_annot --write_html --output_fname_dem_html dem_results.html --seed 666 --n_cpu 20 (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!) Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-10-14T160931.051122.snakemake.log WorkflowError: At least one job did not complete successfully.