Errors with solvePnPRansac on eval_pose

[rhodriguerrier]

Hi @Junyi42,

I have been running the following command for a couple different checkpoints:

CUDA_VISIBLE_DEVICES=0 torchrun --nproc_per_node=1 --master_port=29604 launch.py --mode=eval_pose --pretrained="checkpoint/path/here.pth" --eval_dataset=sintel --output_dir="results/sintel_video_depth" --full_seq --no_crop

However, for some checkpoints the script will error on the cv2.solvePnPRansac function.
I got this error on the 11th sequence when trying Dust3r's checkpoint:

[rank0]:     success, R, T, inliers = cv2.solvePnPRansac(pts3d[msk], pixels[msk], K, None,
[rank0]: cv2.error: OpenCV(4.10.0) /io/opencv/modules/calib3d/src/sqpnp.cpp:242: error: (-215:Assertion failed) point_coordinate_variance >= POINT_VARIANCE_THRESHOLD in function 'computeOmega'

And I got this error on the 15th sequence when trying a checkpoint I trained using just Point Odyssey:

[rank0]:     success, R, T, inliers = cv2.solvePnPRansac(pts3d[msk], pixels[msk], K, None,
[rank0]: cv2.error: OpenCV(4.10.0) /io/opencv/modules/calib3d/src/sqpnp.cpp:264: error: (-215:Assertion failed) ++num_null_vectors_ <= 6 in function 'computeOmega'

Do you know how to fix this?
Any help would be much appreciated.
Thanks

[Junyi42]

Hi @rhodriguerrier,

This is because there are not enough points for PnP algorithm to perform optimization, which is usually because the confidence threshold for generating the msk is too high.

You could set this line: https://github.com/Junyi42/monst3r/blob/main/dust3r/cloud_opt/base_opt.py#L64
min_conf_thr=3, to a lower threshold, e.g., min_conf_thr=1,

Hope this helps!