How many targets can Targets handle?

[myushen]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I have approximately 24k files. Below is a simplified version of the pipeline: It reads the file names from the RDS, loads them dynamically, and then branches over to other targets in subsequent steps. Initially, I processed the pipeline in batches of 2,500 samples each (For instance, firstly RDS has 2500 samples, then increases to 5000, etc). However, during the second batch, dispatching data_object branches took an unusually long time—a process that typically takes just a few seconds. I’m not sure if this issue is related to how crew.cluster manages large increments of targets, or if there is a limit on the number of targets that Targets can dispatch.

library(targets)
library(zellkonverter)
library(crew)
library(crew.cluster)

computing_resources = crew.cluster::crew_controller_slurm(
    slurm_memory_gigabytes_per_cpu = 20,
    slurm_cpus_per_task = 1,
    workers = 50,
    tasks_max = 5,
    verbose = T
  )

tar_option_set(
  controller = computing_resources
  #cue = tar_cue(mode = "never")
)

read_data_container <- function(file,
                                container_type = "anndata"){

  switch(container_type,
         "anndata" = zellkonverter::readH5AD(file, reader = "R", use_hdf5 = TRUE, 
                                             obs = FALSE, raw = FALSE, layers = FALSE),
         "sce_rds" = readRDS(file),
         "seurat_rds" = readRDS(file),
         "sce_hdf5" = loadHDF5SummarizedExperiment(file),
         "seurat_h5" = SeuratDisk::LoadH5Seurat(file)
  )
}


list(
  tar_target(
    file_path, 
    readRDS("input_file.rds") |> as.character() 
  ),
  tar_target(
    data_object,
    read_data_container(file_path, container_type = "anndata"),
    pattern = map(file_path),
    iteration = "list"
  ),
  tar_target(
   # other steps rely on data_object targets
)
)
#tar_make(reporter = "verbose_positives")

This screenshot shows that many branches have been dispatched, but none have completed yet.

[wlandau]

Is the issue that messages like "dispatched branch data_object_dbbcba117fd6da13" and "dispatched branch data_object_7fff03fa2ec02af0" creep along and are slow to print to your R console, or that the underlying work is slow to complete? In the former case, profiling may give us a better idea of what exactly is slowing down execution: https://books.ropensci.org/targets/performance.html#profiling. In the latter case, it could be that your SLURM cluster is busy and all your jobs are waiting in a queue, and monitoring can tell you if your jobs are actually running: https://wlandau.github.io/crew.cluster/index.html#monitoring

[wlandau]

Would updating version help?

Yes, it's worth upgrading nanonext, mirai, crew, and crew.cluster to current CRAN.

I’ve just attempted to cancel the running jobs, but the crew workers still appear to be unable to connect without targets.

Not sure what you mean. If you are manually cancelling running jobs, then there won't be workers to connect.

This is helpful to see:

# A tibble: 1 × 6
  worker online instances assigned complete socket                                          
   <int> <lgl>      <int>    <int>    <int> <chr>                                           
1      1 FALSE          0        0        0 wss://10.8.24.177:40561/ea2df2207147ceecf6d5ff7b

That tells me the SLURM jobs are not able to connect back to the main process. If your compute nodes are firewalled off from the main node running the crew controller, then you might not be able to use crew or clustermq with targets because both packages rely on network connections. The discussions at wlandau/crew.cluster#1 and wlandau/crew.cluster#35 might be relevant.

[multimeric]

To clear things up, crew.cluster::crew_controller_slurm(workers = 1, verbose = TRUE) doesn't work on our cluster because its default memory allocation is 1Mb which is never sufficient. This is not the underlying problem here. I have proposed wlandau/crew#184 which might give us some tools for debugging the idle workers.

[myushen]

I was able to track one of the pids with strace -p before it entered sleep mode, but I’m unsure about the root cause of the issue. Attached the full context of strace log since it exceeds the character limit for comments.
strace_log.txt

[wlandau]

That trace has a lot of brk() calls and some read() calls, which seem to indicate memory-intensive tasks that read from the file system. That's a lot like what your pipeline is supposed to be doing. Maybe this was a worker that successfully ran stuff.

If you're concerned that a worker process is idling when it is supposed to be doing tasks, it is important to confirm that the worker is connected to the main controlling process. online = FALSE from #1329 (reply in thread) is important: the worker can't receive tasks if it is not connected.

[myushen]

The worker can connect to the main controlling process by setting slurm_memory_gigabytes_per_cpu=1 on our cluster.

crew.cluster::crew_controller_slurm(workers = 1, verbose = TRUE,  slurm_memory_gigabytes_per_cpu=1)
# A tibble: 1 × 6
  worker online instances assigned complete socket                                          
   <int> <lgl>      <int>    <int>    <int> <chr>                                           
1      1 TRUE           1        0        0 wss://10.8.24.183:33887/194908090d81eac97b51ed2a

Here, online = TRUE indicates successful connection.

Could the last line in the strace_log clarify why jobs were hanging, particularly the line just before the pid enters sleep mode?