Can I run metaspades up to each major checkpoint?

[AroneyS]

I would like to run metaspades on the Australian National Computing Infrastructure (NCI), but they have a walltime limit of 24h. Some of our assemblies hit that walltime, but perhaps could finish if run as separate commands until checkpoints. Is there a way to run metaspades until it completes a particular checkpoint and then stop, so that it can be continued in a separate job? This way there would be no CPU wastage.

Also, is there a list somewhere of where the checkpoints are? As far as I know, there are checkpoints after read correction and after each kmer, are there others? What checkpoints does --checkpoints all add?

[asl]

Also, is there a list somewhere of where the checkpoints are? As far as I know, there are checkpoints after read correction and after each kmer, are there others? What checkpoints does --checkpoints all add?

There are different checkpoints. Some are "free", so no additional time / disk space would be required. You do not need to do anything special, just running --continue would continue from such a checkpoint.

--checkpoints all (or --checkpoints last) would generate more fine-grained checkpoints. Note that this would increase the overall assembly time and disk requirements as internal state would be saved. Running then --continue or --restart-from last would restart from last saved internal checkpoint.

[asl]

The fine-grained checkpoints are internal thing. They are different depending on the assembly mode, parameters, etc. This is why they are not named. Essentially there is "last one" that is detected automatically. I would not suggest to rely on particular files, etc.

[AroneyS]

What about the primary checkpoints? e.g. completion of a kmer set?

[asl]

It all depends on your job. What if some stage would require more than 24 hours?

[AroneyS]

In that case it wont run on the server so it would be cancelled.
I mainly want a way to know which jobs are worth retrying. So which jobs wont fit completely in 24h, but has checkpoints that each individually do.

I think the best way to do this with minimal wasted usage is to essentially run each checkpoint as a separate job and abandon if any one checkpoint doesn't complete.

[asl]

I would probably suggest you much more simpler approach:

• Monitor which jobs were cancelled due to out of wall-time
• Re-run them using --continue
• If they did not progress (according to the log) – abandon, otherwise repeat --continue

This works with both fine-grained and coarse checkpoints.