Meaning of the annotation_dict entry of CoTracker3_Kubric data

[zw615]

Hi, thanks for the great work!

I just started to play with the newly released CoTracker3_Kubric dataset. However, I found that there are three different entries associated with depth. There are "coords_depth" and "depth" in the annotation_dict (e.g. 0000.npy) of each sample , and there is also a depth folder where it looks like the depth of each frame is stored in different npy files (e.g. 000.npy-119.npy).

My preliminary exploration sort of indicate the annot_dict["depth"] uses the absolute 512x512 depth map, and annot_dict["depth"] stores the relative depth of each tracked point. However, what is the difference between annot_dict["depth"] and the npy fils under the depth folder? Could you please elaborate on the meaning of these entries?

Moreover, I wonder how did you convert the classic Kubric TFDS format into the current format. I see it in your previous post you used traj_annots = {"coords": data["target_points"], "visibility": data["occluded"]}, but the other important entries like depth or camera or other important meta infos are missing. I wonder if you could show how the conversion between TFDS Kubric data and CoTracker3_Kubric data?

Thanks a lot!

[Charles-Lu]

The values in annot_dict["depth"] look similar to the data used here: https://github.com/google-research/kubric/blob/0ee21e2a723b2131123d67e55d1f65b6d0e6cf0f/challenges/point_tracking/dataset.py#L536-L540