Low AP when training on custom data

[Shubhamai]

Hi, thank you for your work on this project! I was trying to train the model on my custom data converted to s3dis format. The AP of one of the instance class chair always remains very low (<0.12) even when overfitting on a single sample.

02/20 21:43:06 - mmengine - INFO - Exp name: oneformer3d_1xb2_s3dis-area-5_20250220_213744
02/20 21:43:06 - mmengine - INFO - Epoch(train) [508][1/1]  lr: 1.2714e-06  eta: 0:00:01  time: 0.2785  data_time: 0.0179  memory: 3346  grad_norm: 27.3595  loss: 2.4154  inst_loss: 2.3841  last_layer_seg_loss: 0.0049  layer_0_seg_loss: 0.0150  layer_1_seg_loss: 0.0062  layer_2_seg_loss: 0.0052
02/20 21:43:06 - mmengine - INFO - Exp name: oneformer3d_1xb2_s3dis-area-5_20250220_213744
02/20 21:43:06 - mmengine - INFO - Epoch(train) [509][1/1]  lr: 9.8136e-07  eta: 0:00:00  time: 0.2782  data_time: 0.0178  memory: 3373  grad_norm: 27.4191  loss: 2.4141  inst_loss: 2.3829  last_layer_seg_loss: 0.0048  layer_0_seg_loss: 0.0150  layer_1_seg_loss: 0.0062  layer_2_seg_loss: 0.0052
02/20 21:43:07 - mmengine - INFO - Exp name: oneformer3d_1xb2_s3dis-area-5_20250220_213744
02/20 21:43:07 - mmengine - INFO - Epoch(train) [510][1/1]  lr: 6.8132e-07  eta: 0:00:00  time: 0.2782  data_time: 0.0179  memory: 3345  grad_norm: 27.4887  loss: 2.4206  inst_loss: 2.3892  last_layer_seg_loss: 0.0049  layer_0_seg_loss: 0.0150  layer_1_seg_loss: 0.0063  layer_2_seg_loss: 0.0052
02/20 21:43:07 - mmengine - INFO - Exp name: oneformer3d_1xb2_s3dis-area-5_20250220_213744
02/20 21:43:07 - mmengine - INFO - Epoch(train) [511][1/1]  lr: 3.6511e-07  eta: 0:00:00  time: 0.2783  data_time: 0.0179  memory: 3345  grad_norm: 27.2968  loss: 2.4147  inst_loss: 2.3833  last_layer_seg_loss: 0.0049  layer_0_seg_loss: 0.0150  layer_1_seg_loss: 0.0063  layer_2_seg_loss: 0.0052
02/20 21:43:07 - mmengine - INFO - Exp name: oneformer3d_1xb2_s3dis-area-5_20250220_213744
02/20 21:43:07 - mmengine - INFO - Epoch(train) [512][1/1]  lr: 0.0000e+00  eta: 0:00:00  time: 0.2788  data_time: 0.0179  memory: 3347  grad_norm: 27.1049  loss: 2.4133  inst_loss: 2.3821  last_layer_seg_loss: 0.0049  layer_0_seg_loss: 0.0150  layer_1_seg_loss: 0.0063  layer_2_seg_loss: 0.0052
02/20 21:43:07 - mmengine - INFO - Saving checkpoint at 512 epochs
02/20 21:43:09 - mmengine - INFO - |        |   IoU   |   PQ   |   RQ   |  SQ   |
02/20 21:43:09 - mmengine - INFO - |all     | 0.1122 | 0.1079 | 0.1600 | 0.1785 |
02/20 21:43:09 - mmengine - INFO - |ceiling | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
02/20 21:43:09 - mmengine - INFO - |floor   | 0.7737 | 0.7737 | 1.0000 | 0.7737 |
02/20 21:43:09 - mmengine - INFO - |wall    | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
02/20 21:43:09 - mmengine - INFO - |beam    | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
02/20 21:43:09 - mmengine - INFO - |column  | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
02/20 21:43:09 - mmengine - INFO - |window  | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
02/20 21:43:09 - mmengine - INFO - |door    | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
02/20 21:43:09 - mmengine - INFO - |table   | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
02/20 21:43:09 - mmengine - INFO - |chair   | 0.1358 | 0.0798 | 0.0800 | 0.9974 |
02/20 21:43:09 - mmengine - INFO - |sofa    | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
02/20 21:43:09 - mmengine - INFO - |bookcase| 0.0000 | 0.0000 | 0.0000 | 0.0000 |
02/20 21:43:09 - mmengine - INFO - |board   | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
02/20 21:43:09 - mmengine - INFO - |clutter | 0.5497 | 0.5497 | 1.0000 | 0.5497 |
02/20 21:43:09 - mmengine - INFO - |nlabeled| 0.0000 | 0.0000 | 0.0000 | 0.0000 |
02/20 21:43:09 - mmengine - INFO - True Positive: 
02/20 21:43:09 - mmengine - INFO - 0    |       1       |       0       |       0       |       0       |       0       |       0       |       0       |       5       |   0|       0       |       0       |       1       |       0
False Positive: 
02/20 21:43:09 - mmengine - INFO - 1    |       0       |       0       |       0       |       1       |       1       |       1       |       0       |       0       |   0|       0       |       0       |       0       |       0
False Negative: 
02/20 21:43:09 - mmengine - INFO - 0    |       0       |       0       |       0       |       0       |       0       |       0       |       0       |       115     |   0|       0       |       0       |       0       |       0
/pipeline/segmentation-v2/oneformer3d/oneformer3d-env/lib/python3.10/site-packages/mmdet3d/evaluation/functional/seg_eval.py:39: RuntimeWarning: invalid value encountered in divide
  return np.diag(hist) / (hist.sum(1) + hist.sum(0) - np.diag(hist))
/pipeline/segmentation-v2/oneformer3d/oneformer3d-env/lib/python3.10/site-packages/mmdet3d/evaluation/functional/seg_eval.py:67: RuntimeWarning: invalid value encountered in divide
  return np.nanmean(np.diag(hist) / hist.sum(axis=1))
02/20 21:43:09 - mmengine - INFO - 
+---------+---------+--------+------+------+--------+--------+------+-------+--------+------+----------+-------+---------+-----------+--------+--------+---------+
| classes | ceiling | floor  | wall | beam | column | window | door | table | chair  | sofa | bookcase | board | clutter | unlabeled | miou   | acc    | acc_cls |
+---------+---------+--------+------+------+--------+--------+------+-------+--------+------+----------+-------+---------+-----------+--------+--------+---------+
| results | nan     | 0.9937 | nan  | nan  | nan    | nan    | nan  | nan   | 0.9917 | nan  | nan      | nan   | 0.9920  | nan       | 0.9925 | 0.9960 | 0.9958  |
+---------+---------+--------+------+------+--------+--------+------+-------+--------+------+----------+-------+---------+-----------+--------+--------+---------+
02/20 21:43:09 - mmengine - INFO - 
+----------+---------+---------+--------+-----------+----------+
| classes  | AP_0.25 | AP_0.50 | AP     | Prec_0.50 | Rec_0.50 |
+----------+---------+---------+--------+-----------+----------+
| ceiling  | nan     | nan     | nan    | nan       | nan      |
| floor    | 1.0000  | 1.0000  | 1.0000 | 1.0000    | 1.0000   |
| wall     | nan     | nan     | nan    | nan       | nan      |
| beam     | nan     | nan     | nan    | nan       | nan      |
| column   | nan     | nan     | nan    | nan       | nan      |
| window   | nan     | nan     | nan    | nan       | nan      |
| door     | nan     | nan     | nan    | nan       | nan      |
| table    | nan     | nan     | nan    | nan       | nan      |
| chair    | 0.1420  | 0.1241  | 0.1102 | 0.5714    | 0.1356   |
| sofa     | nan     | nan     | nan    | nan       | nan      |
| bookcase | nan     | nan     | nan    | nan       | nan      |
| board    | nan     | nan     | nan    | nan       | nan      |
| clutter  | 1.0000  | 1.0000  | 1.0000 | 1.0000    | 1.0000   |
+----------+---------+---------+--------+-----------+----------+
| Overall  | 0.7140  | 0.7080  | 0.7034 | 0.8571    | 0.7119   |
+----------+---------+---------+--------+-----------+----------+
02/20 21:43:09 - mmengine - INFO - Epoch(val) [512][1/1]    miou: 0.9925  all_ap: 0.7034  all_ap_50%: 0.7080  all_ap_25%: 0.7140  pq: 0.1079  data_time: 0.0125  time: 0.1936

Replicating the issue

1. Simply download s3dis_custom from this google drive link and change the data_root in config accordingly.

2. Also change the following lines:

train_area = [1]
test_area = 1

No other changes are required. I also uploaded Area_1.ply as the training sample represented in s3dis_custom.

To rule out any other bugs, I also tried training on original s3dis (s3dis_original on drive) on a single sample and the training worked perfectly!

---

I tried debugging several different ways and changing several things from input points normalization and changing things as little as possible to keep the original method of processing s3dis data. But none of the changes produced any good AP.

I would love your help on what should I be debugging next to fix this issue. I am uncertain whether this is due to an architecture limitation or a simple data preprocessing bug. If it's an architecture limitation, what changes I can make (config or oneformer3d.py) to improve the results ?

Thanks again