Massively parallel scripting

[rknx]

For cross validation phase:

• Transfer dataset to remote location (Hipergator)
• install packages parallel, foreach and rslurm
• in a script
  • make functions to execute
  • make parameter list (one per slurm submission)
  • set slurm options
  • execute
• copy rda back

[rknx]

http://cyberhelp.sesync.org/rslurm/articles/rslurm.html

[rknx]

How to tell how much time it is going to take?

How to progress how much job is completed?

• print progress to stdout?

One big slurm batch with large allocation or small multiple submissions?

rslurm can break a loop up into small jobs automatically, over multiple nodes.

• so, yes, multiple jobs