Cannot collect data in task 04

[xiedun2001]

Dear developers,

I performed task 01 and task 04 in a Supercomputing Center. The task 01 finished correctly. It created 5 jobs in task 04. The dogen.log shows only 1 job finished.

I checked the files in Supercomputing Center, found other 4 jobs also finished, but somehow the dpgen did not recognize it.

It reported "status command squeue fails to execute" in nohup.out.

This is my machine.json. How to deal with this problem?

[AnguseZhang]

1. Could you list the five folders in "work_path_dnn" via ls * and see whether there are files called "xxxxx_tag_finished"?
2. Can DP-GEN automatically restart by dpgen run param.json machine.json?

[xiedun2001]

1. All of the 4 folders have xxxxx_tag_finished file.
   
   
   
   
2. I tried dpgen run param.json machine.json. It shows:
   
   

[AnguseZhang]

Could you try to execute command squeue -o "%.18i %.2t" -j 30279 and see what is printed?

[xiedun2001]

The job is not in the sequence.

[felix5572]

Maybe because of slurm partition settings?

How to query the job information in your cluster

[xiedun2001]

I use parajobs to query the job which is running and in queue.

[xiedun2001]

I reduced the number of tasks from 2500 to 500, and reduce the md step from 3000 to 1000. The task 4 finished correctly. Dose the number and the size of result files in task 4 have a limit?

[felix5572]

Probably not. Maybe you could edit the source yourself, and print and debug the relating information yourself?

[njzjz]

I am wondering if the error was printed to stdout but not stderr. You can try squeue -o "%.18i %.2t" -j 30279 1>/dev/null and squeue -o "%.18i %.2t" -j 30279 2>dev/null to see if there is any difference.