DPGEN: 'Get error code %d in locally calling %s with job: %s' when using submission script (PBS scheduler)

[srimayee-27-jnc]

Hi,
I am receiving the following error when submitting jobs using submission script for PBS scheduler. Kindly help me diagnose the problem. Given below is the error file:

Traceback (most recent call last):
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpdispatcher/submission.py", line 358, in handle_unexpected_submission_state
    job.handle_unexpected_job_state()
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpdispatcher/submission.py", line 863, in handle_unexpected_job_state
    self.submit_job()
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpdispatcher/submission.py", line 926, in submit_job
    job_id = self.machine.do_submit(self)
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpdispatcher/machines/pbs.py", line 59, in do_submit
    stdin, stdout, stderr = self.context.block_checkcall(
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpdispatcher/contexts/local_context.py", line 294, in block_checkcall
    raise RuntimeError(
RuntimeError: ('Get error code %d in locally calling %s with job: %s ', (1, 'cd /home/srimayee/dpgen_test/cvd/MoS2/trial5/run/a872398fad5fd56641919e04bd1c3a5e63a30b2c && qsub e3c45ca96ae897f2d4699aa468e2a8f63a8986b4.sub', 'a872398fad5fd56641919e04bd1c3a5e63a30b2c'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/srimayee/.conda/envs/dpgen/bin/dpgen", line 8, in <module>
    sys.exit(main())
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpgen/main.py", line 255, in main
    args.func(args)
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpgen/generator/run.py", line 5411, in gen_run
    run_iter(args.PARAM, args.MACHINE)
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpgen/generator/run.py", line 4763, in run_iter
    run_fp(ii, jdata, mdata)
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpgen/generator/run.py", line 3985, in run_fp
    run_fp_inner(
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpgen/generator/run.py", line 3964, in run_fp_inner
    submission.run_submission()
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpdispatcher/submission.py", line 231, in run_submission
    self.handle_unexpected_submission_state()
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpdispatcher/submission.py", line 362, in handle_unexpected_submission_state
    raise RuntimeError(
RuntimeError: Meet errors will handle unexpected submission state.
Debug information: remote_root==/home/srimayee/dpgen_test/cvd/MoS2/trial5/run/a872398fad5fd56641919e04bd1c3a5e63a30b2c.
Debug information: submission_hash==a872398fad5fd56641919e04bd1c3a5e63a30b2c.
Please check error messages above and in remote_root. The submission information is saved in /home/srimayee/.dpdispatcher/submission/a872398fad5fd56641919e04bd1c3a5e63a30b2c.json.
For furthur actions, run the following command with proper flags: dpdisp submission a872398fad5fd56641919e04bd1c3a5e63a30b2c
Traceback (most recent call last):
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpdispatcher/submission.py", line 358, in handle_unexpected_submission_state
    job.handle_unexpected_job_state()
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpdispatcher/submission.py", line 863, in handle_unexpected_job_state
    self.submit_job()
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpdispatcher/submission.py", line 926, in submit_job
    job_id = self.machine.do_submit(self)
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpdispatcher/machines/pbs.py", line 59, in do_submit
    stdin, stdout, stderr = self.context.block_checkcall(
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpdispatcher/contexts/local_context.py", line 294, in block_checkcall
    raise RuntimeError(
RuntimeError: ('Get error code %d in locally calling %s with job: %s ', (175, 'cd /home/srimayee/dpgen_test/cvd/MoS2/trial5/run/a872398fad5fd56641919e04bd1c3a5e63a30b2c && qsub e3c45ca96ae897f2d4699aa468e2a8f63a8986b4.sub', 'a872398fad5fd56641919e04bd1c3a5e63a30b2c'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/srimayee/.conda/envs/dpgen/bin/dpgen", line 8, in <module>
    sys.exit(main())
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpgen/main.py", line 255, in main
    args.func(args)
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpgen/generator/run.py", line 5411, in gen_run
    run_iter(args.PARAM, args.MACHINE)
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpgen/generator/run.py", line 4763, in run_iter
    run_fp(ii, jdata, mdata)
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpgen/generator/run.py", line 3985, in run_fp
    run_fp_inner(
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpgen/generator/run.py", line 3964, in run_fp_inner
    submission.run_submission()
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpdispatcher/submission.py", line 231, in run_submission
    self.handle_unexpected_submission_state()
  File "/home/srimayee/.conda/envs/dpgen/lib/python3.9/site-packages/dpdispatcher/submission.py", line 362, in handle_unexpected_submission_state
    raise RuntimeError(
RuntimeError: Meet errors will handle unexpected submission state.
Debug information: remote_root==/home/srimayee/dpgen_test/cvd/MoS2/trial5/run/a872398fad5fd56641919e04bd1c3a5e63a30b2c.
Debug information: submission_hash==a872398fad5fd56641919e04bd1c3a5e63a30b2c.
Please check error messages above and in remote_root. The submission information is saved in /home/srimayee/.dpdispatcher/submission/a872398fad5fd56641919e04bd1c3a5e63a30b2c.json.
For furthur actions, run the following command 

Given below is the submission script:

#!/bin/sh
#PBS -N dpgen
#PBS -q AGR
#PBS -l select=1:ncpus=48:host=node12-ib+1:ncpus=48:host=node17-ib
#PBS -l walltime=150:00:00
#PBS -l place=scatter

cd $PBS_O_WORKDIR

module load vasp-VTST-intel

module load python-3.9

source ~/deepmd_source.sh

mpirun -np 2 ~/.conda/envs/dpgen/bin/dpgen run param.json machine.json

[njzjz]

Do you get any error when executing cd /home/srimayee/dpgen_test/cvd/MoS2/trial5/run/a872398fad5fd56641919e04bd1c3a5e63a30b2c && qsub e3c45ca96ae897f2d4699aa468e2a8f63a8986b4.sub manually?

The error message seems unclear here.

[srimayee-27-jnc]

Hi, thank you for your response. We executed the suggested commands. We see the following files were created

1. e3c45ca96ae897f2d4699aa468e2a8f63a8986b4.sub.o67689..................empty
2. e3c45ca96ae897f2d4699aa468e2a8f63a8986b4_flag_if_job_task_fail .................contains "0"
3. e3c45ca96ae897f2d4699aa468e2a8f63a8986b4_job_tag_finished..............empty

This suggests to us that the job failed. Given below is e3c45ca96ae897f2d4699aa468e2a8f63a8986b4.sub:

#!/bin/bash -l
#PBS -l select=1:ncpus=24
#PBS -j oe
#PBS -q AGR

#PBS -l walltime=150:00:00
#PBS -l select=1:ncpus=48:host=node12-ib+1:ncpus=48:host=node17-ib
#PBS -l place=scatter
cd $PBS_O_WORKDIR


REMOTE_ROOT=$(readlink -f /home/srimayee/dpgen_test/cvd/MoS2/trial5/run/a872398fad5fd56641919e04bd1c3a5e63a30b2c)
echo 0 > $REMOTE_ROOT/e3c45ca96ae897f2d4699aa468e2a8f63a8986b4_flag_if_job_task_fail
test $? -ne 0 && exit 1







source $REMOTE_ROOT/e3c45ca96ae897f2d4699aa468e2a8f63a8986b4.sub.run

cd $REMOTE_ROOT
test $? -ne 0 && exit 1

wait
FLAG_IF_JOB_TASK_FAIL=$(cat e3c45ca96ae897f2d4699aa468e2a8f63a8986b4_flag_if_job_task_fail)
if test $FLAG_IF_JOB_TASK_FAIL -eq 0; then touch e3c45ca96ae897f2d4699aa468e2a8f63a8986b4_job_tag_finished; else exit 1;fi

Kindly advise. Thanks and regards.