Tips: Getting tar_make_future stable on Slurm

[stuvet]

I had a nightmare getting this working. I'm going to summarise my last few weeks below to help others who may follow. That probably includes myself in about 2 days.

Problem Infrastructure

• Scalable cluster on GCP with 0 resting-state workers in some partitions, and with most partitions using GCP-preemptible workers (I think the same would be true for slurm-preempted jobs).
• I have sole custody, so I could do some antisocial things to track down some of these problems.
• My most common issues below would be seen far less often on larger clusters with resources always available & no hardware preemption.

Errors

Fatal Pipeline Errors

• Errors in the backend infrastructure (e.g. preemption, or queueing for a long time when resources aren't available, or hitting job walltime) take down the entire pipeline, even when error='continue'

Log file not available when some jobs start running

• Note: .log file, not .rds (below)

Error : Log file ... for job with id 1 not available

BatchtoolsExpiration

• e.g. when jobs are allocated resources after queuing, or are requeued after resource preemption.
• The job may or may not run & complete normally on Slurm, but you receive an 'expired' error back in targets. The job may be left running & 'orphaned' for long after you receive the error.

##(contrived example - this one is normal - job hit walltime)

Error : BatchtoolsExpiration: Future ('job_name) expired (registry path /<my_working_directory>/.future/<date_time_uid>/<job_name>).. The last few lines of the logged output:
### [bt]: This is batchtools v0.9.16
### [bt]: Starting calculation of 1 jobs
### [bt]: Setting working directory to '/<my_working_directory>'
Loading required package: targets
### [bt]: Memory measurement disabled
### [bt]: Starting job [batchtools job.id=1]
### [bt]: Setting seed to 4903 ...
slurmstepd: error: *** JOB 21339 ON <my_slurm_controller_node> CANCELLED AT 2021-07-23T08:49:55 DUE TO TIME LIMIT ***

Gzfile - cannot open the connection

• Note: .rds file, not .log (above)

Error in gzfile(file, "rb") : cannot open the connection
Calls: <Anonymous> -> doJobCollection.character -> readRDS -> gzfile
In addition: Warning message:
In gzfile(file, "rb") :
  cannot open compressed file '<my_working_dir>/.future/<date_time_uid>/<jobname_uid>/jobs/job920878958e6077e5362e4c93d33312f4.rds', probable reason 'No such file or directory'

I'll post my answers below.

[stuvet]

Solutions

Fatal Pipeline Errors

• Update Targets

  • 141859f or after to avoid infrastructure problems & upstream bugs taking down the whole pipeline.

Log File Not Available

• Note: .log file, not .rds (below)

• If this happens as the worker starts up (or as jobs are moved to workers after being queued), you may need to patch batchtools to avoid this bug Bugfix in waitForFile mllg/batchtools#276 (comment)

BatchtoolsExpiration

• If your pipeline is taken down by jobs hitting walltime, go back to ⬆️ ⬆️ 😉
  • If you're using future.batchtools:::batchtools_slurm in your future::plan you need to check the batchtools function it refers to is correctly mapping all possible slurm job state codes.
  • Any slurm job state left unhandled will be interpreted as expired
  • Either configure & use future.batchtools:::batchtools_custom in your plan instead, or adjust batchtools like Mapped missing Slurm job state codes mllg/batchtools#277 (comment)

Gzfile - cannot open the connection

• Note: .rds file, not .log (above)

• AFAIK this only happened to me after doing all of the above, particularly after mapping the 'RQ' (Requeued) status code to 'queued' (previously it triggered 'expired')

• Not yet solved, but there is a workaround for now - just pass resources$chunks.as.arrayjobs = TRUE. This doesn't make proper HPC array jobs because they're submitted singly by future.batchtools, but it stops the file being deleted at the start of the first (failed) run. future.batchtools cleans up the files afterwards anyway.

Summary

• As of today, with all the above fixes Slurm submission, queuing, machine/resource preemption and requeuing are all handled properly by slurm+batchtools+future.batchtools+targets.

Hope it helps someone!

[sakrejda-umich]

This is awesome, thanks for sharing!