./tools/dist_test.sh is too slow

[zhangluustb]

./tools/dist_test.sh在dim模型test时特别慢，不论设置OMP_NUM_THREADS还是torch设置threads 都会卡住

[zhangluustb]

@lindahua @hellock

[hejm37]

Can you provide more details on your test case?

• What command do you use to run the test?
• What is the speed of the program (like how many images per second)?
• What do you mean by "get stuck"? Is the program not running at all?

[zhangluustb]

Can you provide more details on your test case?

• What command do you use to run the test?
• What is the speed of the program (like how many images per second)?
• What do you mean by "get stuck"? Is the program not running at all?

1. ./tools/dist_test.sh configs/mattors/dim/dim_stage1_v16....py ?.pth 2(evaluation=dict(interval=4000,save_image=False))
2. 1 task/s
3. not running at all 最后中断或没法进行evaluate

同时，在./tools/dist_train.sh configs/mattors/dim/dim_stage1_v16....py 2 也会出现这个问题，evalutation卡住无法进行。
因为验证集为1000例，大概跑到400例时便会越来越慢，直到卡住。
可以加你们的QQ或者微信群进行交流求助吗？多谢
@hejm37

[hejm37]

The correct usage of the dist_test.sh is

./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM} [--out ${RESULT_FILE}] [--save-path ${IMAGE_SAVE_PATH}]

Do you pass ${CHECKPOINT_FILE} to the program?
Can you check the CPU usage (through top/htop) and the GPU usage (through nvidia-smi) when the program get stuck?

You can join the QQ group of OpenMMLab community: 144762544.

In addition, you should use English if possible since others may encounter the same problem.

[zhangluustb]

Thank you for your reply. There is no error in the my command input. The GPU were about 10800MiB/11177MiB，but no Volatile CPU-Util was used; The CPUs were only filled 2 in 6 cores. I will post the htop and nvidai-smi picture when I reproduce the error.

[cnnAndBn]

@zhangluustb,hi ,I also encountered this problem, have you sloved it? if so ,how to do, thanks

[innerlee]

Please monitor the cpu memory usage to see if they are full.

[cnnAndBn]

@innerlee I notice the memory is occupied a lot nearly 95% (16G for me), and even may be broken down