Run GSEA hanging for hours

[rogercasalsfr]

Hi everyone,

I am struggling to get the last step running scenicplus which is when is GSEA running.

My scplus_obj is:

        metadata_regions:'Chromosome', 'Start', 'End', 'Width', 'cisTopic_nr_frag', 'cisTopic_log_nr_frag', 'cisTopic_nr_acc', 'cisTopic_log_nr_acc'
        metadata_cell:'GEX_obs_id', 'GEX_cell_type', 'GEX_donor_id', 'GEX_louvain', 'ACC_cisTopic_nr_frag', 'ACC_cisTopic_log_nr_frag', 'ACC_cisTopic_nr_acc', 'ACC_cisTopic_log_nr_acc', 'ACC_sample_id', 'ACC_cell_type'
        menr:'CTX_topics_otsu_All', 'CTX_topics_otsu_No_promoters', 'DEM_topics_otsu_All', 'DEM_topics_otsu_No_promoters', 'CTX_topics_top_3_All', 'CTX_topics_top_3_No_promoters', 'DEM_topics_top_3_All', 'DEM_topics_top_3_No_promoters', 'CTX_DARs_All', 'CTX_DARs_No_promoters', 'DEM_DARs_All', 'DEM_DARs_No_promoters'
        dr_cell:'GEX_X_pca', 'GEX_X_umap'

I'm running the function in an HPC, locally.

My code is this:

run_scenicplus(
        scplus_obj = scplus_obj,
        variable = ['GEX_celltype'],
        species = 'hsapiens',
        assembly = 'hg38',
        tf_file = '/path/TF_names_v_1.01.txt',
        save_path = os.path.join('/path_to_object/scplus_obj2.pkl'),
        #biomart_host = biomart_host,
        upstream = [1000, 150000],
        downstream = [1000, 150000],
        calculate_TF_eGRN_correlation = True,
        calculate_DEGs_DARs = True,
        export_to_loom_file = True,
        export_to_UCSC_file = False,
        path_bedToBigBed = '/path_to_Big_Bed',
        n_cpu = 16,
        _temp_dir = '/path_to_tmp_dir')

I already have all the outputs:

2023-12-18 11:27:59,744 R2G          INFO     Took 1223.3278999328613 seconds
2023-12-18 11:27:59,747 R2G          INFO     Calculating region to gene correlation, using SR method
2023-12-18 11:38:46,592 R2G          INFO     Took 646.8447403907776 seconds
2023-12-18 11:39:04,790 R2G          INFO     Done!
2023-12-18 11:39:05,064 SCENIC+_wrapper INFO     Inferring TF to gene relationships
2023-12-18 11:39:14,411 TF2G         INFO     Calculating TF to gene correlation, using GBM method
2023-12-18 14:18:31,312 TF2G         INFO     Took 9556.900450229645 seconds
2023-12-18 14:18:31,332 TF2G         INFO     Adding correlation coefficients to adjacencies.
2023-12-18 14:19:35,223 TF2G         INFO     Warning: adding TFs as their own target to adjecencies matrix. Importance values will be max + 1e-05
2023-12-18 14:19:42,697 TF2G         INFO     Adding importance x rho scores to adjacencies.
2023-12-18 14:19:42,720 TF2G         INFO     Took 71.38793706893921 seconds
2023-12-18 14:19:43,015 SCENIC+_wrapper INFO     Build eGRN
2023-12-18 14:19:43,016 GSEA         INFO     Thresholding region to gene relationships
2023-12-18 14:35:57,730 GSEA         INFO     Subsetting TF2G adjacencies for TF with motif.
2023-12-18 14:36:05,289 GSEA         INFO     Running GSEA...

But then I get this error:

^[[36m(_ray_run_gsea_for_e_module pid=27911)^[[0m   norm_tag =  1.0/sum_correl_tag
^[[36m(_ray_run_gsea_for_e_module pid=27911)^[[0m /storage/projects/uvic24/scenicplus_env/lib/python3.8/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply
^[[36m(_ray_run_gsea_for_e_module pid=27911)^[[0m   RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) 

initializing:   1%|          | 225/22065 [00:40<1:09:56,  5.20it/s]^Minitializing:   1%|          | 226/22065 [00:40<1:12:12,  5.04it/s]^Minitializing:   1%|          | 227/22065 [00:40<1:15:40,  4.81it/s]^Minitializing:   1%|          | 228/22065 [00:40<1:14:12,  4.90it/s]^Minitializing:   1%|          | 229/22065 [00:40<1:13:35,  4.94it/s]^Minitializing:   1%|          | 230/22065 [00:41<1:11:12,  5.11it/s]^Minitializing:   1%|          | 231/22065 [00:41<1:06:44,  5.45it/s]^Minitializing:   1%|          | 232/22065 [00:41<1:05:32,  5.55it/s]^Minitializing:   1%|          | 233/22065 [00:41<1:05:08,  5.59it/s]^Minitializing:   1%|          | 234/22065 [00:41<1:05:42,  5.54it/s]^Minitializing:   1%|          | 235/22065 [00:41<1:05:07,  5.59it/s]^Minitializing:   1%|          | 236/22065 [00:42<1:05:40,  5.54it/s]^Minitializing:   1%|          | 237/22065

And then I wait for hours for the GSEA step to finish and it doesn't finish.
Has anybody got the same error, or know how to solve it?

Thanks in advance.

[SeppeDeWinter]

Hi @rogercasalsfr

I would suggest to use the code in the development branch. That one is a lot faster.
See: #202 for more information.

All the best,

Seppe