Low CPU usage in dpgen

[XiaMinqi]

I tried to submit the job by shell on local machine. The CPU usage is quite low somehow.
It seems that the sum usage of each task is just %CPU = 100%. That means only about one physical core was used.

The CPU usage at train:

The CPU usage at fp:

The CPU usage at model_devi:

The lammps log file shows:

Performance: 13.931 ns/day, 1.723 hours/ns, 80.622 timesteps/s
47.2% CPU use with 1 MPI tasks x 8 OpenMP threads

The VASP OUTCAR shows:

 vasp.6.3.0 20Jan22 (build Oct 29 2022 15:50:47) complex                        
  
 executed on             LinuxIFC date 2022.12.15  22:59:27
 running on    1 total cores
 distrk:  each k-point on    1 cores,    1 groups
 distr:  one band on NCORE=   1 cores,    1 groups

The above information suggests the task did't make full use of CPU resources.

The parameters in my machine.json file is:

{
  "api_version": "1.0",
  "deepmd_version": "2.1.5",
  "train" :
    {
      "command": "dp",
      "machine": {
        "batch_type": "shell",
	"_machine_type": "shell",
        "context_type": "local",
        "local_root" : "./",
	"remote_root": "/home/lijun/xmq/12.1/dpgen_example/work",
        "remote_profile":{
		"hostname" :"ljc9"
        }
      },
      "resources": {
        "number_node": 1,
        "local_root":"./",
        "cpu_per_node": 56,
        "gpu_per_node": 0,
        "_queue_name": "GPU",
        "group_size": 1
      }
    },
  "model_devi":
    {
      "command": "lmp -i input.lammps -v restart 0",
      "machine": {
        "batch_type": "shell",
        "context_type": "local",
        "local_root" : "./",
	"remote_root": "/home/lijun/xmq/12.1/dpgen_example/work",
        "remote_profile":{
	"hostname" :"ljc9"
        }
      },
      "resources": {
        "number_node": 1,
        "local_root":"./",
        "cpu_per_node": 56,
        "gpu_per_node": 0,
        "_queue_name": "GPU",
        "group_size": 5
      }
    },
  "fp":
    {
      "command": "vasp_std",
      "machine": {
        "batch_type": "shell",
        "context_type": "local",
        "local_root" : "./",
	"remote_root": "/home/lijun/xmq/12.1/dpgen_example/work",
        "remote_profile":{
	"hostname" :"ljc9"
        }
      },
      "resources": {
        "number_node": 1,
        "cpu_per_node": 56,
        "gpu_per_node": 0,
        "_queue_name": "CPU",
        "group_size": 20,
        "local_root":"./",
	"source_list": ["/opt/intel/oneapi/setvars.sh"]
      }
    }
}

At the meantime I have set the environment variables according to the manual as:

export OMP_NUM_THREADS=8
export TF_INTRA_OP_PARALLELISM_THREADS=8
export TF_INTER_OP_PARALLELISM_THREADS=7

I tried to add "mpirun -np 56" in machine.json before each command and that did not solve the problem: the number of threads would increase while the cpu usage of each thread would drop to 0.9%.
Hope advice on how to improve the efficiency to reach the best performance. :)

[njzjz]

If you are running multiple tasks simultaneously, do not use all the resources for each task. To run only one task at a time, set group_size to 0.

[XiaMinqi]

I got it, thank you for your kind suggestion!