CodeCamp #106: Support SOLOv2 deployment with OpenVINO (#1521)

* support solov2 openvino

* fix lint

* Update benchmark.md

* Update __init__.py

* fix names and delete redundant codes

* fix names and redundant codes

* fix indent

* fix lint

* fix lint

* delete 'ctx' arguments

* [Fix]fix lint

* 修改：     mmdeploy/codebase/mmdet/models/dense_heads/solov2_head.py
	修改：     mmdeploy/codebase/mmdet/models/detectors/single_stage_instance_seg.py

* Delete matrix_nms.py

* Update __init__.py

* Update linspace.py

* Update solov2_head.py

* Update solov2_head.py

* update

* fix num_groups

* update docs

* add end2endmodel for solo

* add end2endmodel for solo

* [Feature] Support End2EndModelSOLO

* Update object_detection_model.py

* add model_cfg to End2EndModel

* add model_cfg to End2EndModel

* Fix bugs for End2EndModel

* fix bugs for End2EndModel

* update docs.

* update conmment

* 更新 benchmark.md

* update regression tests

* fix errors in docs

docs/en/03-benchmark/benchmark.md

@@ -613,6 +613,7 @@ Users can directly test the performance through [how_to_evaluate_a_model.md](../
     <td align="center">81.18</td>
     <td align="center">-</td>
     <td align="center">-</td>
+    <td align="center">-</td>
   </tr>
   <tr>
     <td align="center">top-5</td>
@@ -623,6 +624,7 @@ Users can directly test the performance through [how_to_evaluate_a_model.md](../
     <td align="center">95.61</td>
     <td align="center">-</td>
     <td align="center">-</td>
+    <td align="center">-</td>
   </tr>
 </tbody>
 </table>
@@ -907,20 +909,35 @@ Users can directly test the performance through [how_to_evaluate_a_model.md](../
     <td align="center">-</td>
   </tr>
   <tr>
-    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo">SOLO</a></td>
-    <td align="center" rowspan="2">Instance Segmentation</td>
-    <td align="center" rowspan="2">COCO2017</td>
+    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo/solo_r50_fpn_1x_coco.py">SOLO</a></td>
+    <td align="center">Instance Segmentation</td>
+    <td align="center">COCO2017</td>
     <td align="center">mask AP</td>
     <td align="center">33.1</td>
     <td align="center">-</td>
-    <td align="center">-</td>
+    <td align="center">32.7</td>
     <td align="center">-</td>
     <td align="center">-</td>
     <td align="center">-</td>
     <td align="center">-</td>
     <td align="center">-</td>
     <td align="center">32.7</td>
   </tr>
+  <tr>
+    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2/solov2_r50_fpn_1x_coco.py">SOLOv2</a></td>
+    <td align="center">Instance Segmentation</td>
+    <td align="center">COCO2017</td>
+    <td align="center">mask AP</td>
+    <td align="center">34.8</td>
+    <td align="center">-</td>
+    <td align="center">34.5</td>
+    <td align="center">-</td>
+    <td align="center">-</td>
+    <td align="center">-</td>
+    <td align="center">-</td>
+    <td align="center">-</td>
+    <td align="center">34.5</td>
+  </tr>
 </tbody>
 </table>
 </div>

docs/en/03-benchmark/supported_models.md

@@ -22,7 +22,8 @@ The table below lists the models that are guaranteed to be exportable to other b
 | [RepPoints](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/reppoints)                       | MMDetection      |      N      |      N      |    Y     |  N   |   ?   |    Y     |   N    |  N   |
 | [DETR](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/detr)                                 | MMDetection      |      N      |      Y      |    Y     |  N   |   ?   |    N     |   N    |  N   |
 | [CenterNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/centernet)                       | MMDetection      |      N      |      Y      |    Y     |  N   |   ?   |    N     |   N    |  N   |
-| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo)                                 | MMDetection      |      N      |      N      |    N     |  N   |   N   |    Y     |   N    |  N   |
+| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo)                                 | MMDetection      |      N      |      Y      |    N     |  N   |   N   |    Y     |   N    |  N   |
+| [SOLOv2](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2)                             | MMDetection      |      N      |      Y      |    N     |  N   |   N   |    Y     |   N    |  N   |
 | [ResNet](https://github.com/open-mmlab/mmclassification/tree/1.x/configs/resnet)                        | MMClassification |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |
 | [ResNeXt](https://github.com/open-mmlab/mmclassification/tree/1.x/configs/resnext)                      | MMClassification |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |
 | [SE-ResNet](https://github.com/open-mmlab/mmclassification/tree/1.x/configs/seresnet)                   | MMClassification |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |

docs/en/04-supported-codebases/mmdet.md

@@ -210,4 +210,5 @@ Besides python API, mmdeploy SDK also provides other FFI (Foreign Function Inter
 | [Cascade Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn) | Instance Segmentation |      Y      |    N     |  N   |   N   |    Y     |
 |      [Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/mask_rcnn)       | Instance Segmentation |      Y      |    Y     |  N   |   N   |    Y     |
 |      [Swin Transformer](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/swin)      | Instance Segmentation |      Y      |    Y     |  N   |   N   |    N     |
-|            [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo)            | Instance Segmentation |      N      |    N     |  N   |   N   |    Y     |
+|            [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo)            | Instance Segmentation |      Y      |    N     |  N   |   N   |    Y     |
+|          [SOLOv2](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2)          | Instance Segmentation |      Y      |    N     |  N   |   N   |    Y     |

docs/zh_cn/03-benchmark/benchmark.md

@@ -633,7 +633,7 @@ GPU: ncnn, TensorRT, PPLNN
     <th align="center" colspan="3">TensorRT</th>
     <th align="center">PPLNN</th>
     <th align="center">Ascend</th>
-    <th algin="center">OpenVINO</th>
+    <th align="center">OpenVINO</th>
   </tr>
 </thead>
 <tbody>
@@ -902,20 +902,35 @@ GPU: ncnn, TensorRT, PPLNN
     <td align="center">-</td>
   </tr>
   <tr>
-    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo">SOLO</a></td>
-    <td align="center" rowspan="2">Instance Segmentation</td>
-    <td align="center" rowspan="2">COCO2017</td>
+    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo/solo_r50_fpn_1x_coco.py">SOLO</a></td>
+    <td align="center">Instance Segmentation</td>
+    <td align="center">COCO2017</td>
     <td align="center">mask AP</td>
     <td align="center">33.1</td>
     <td align="center">-</td>
-    <td align="center">-</td>
+    <td align="center">32.7</td>
     <td align="center">-</td>
     <td align="center">-</td>
     <td align="center">-</td>
     <td align="center">-</td>
     <td align="center">-</td>
     <td align="center">32.7</td>
   </tr>
+  <tr>
+    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2/solov2_r50_fpn_1x_coco.py">SOLOv2</a></td>
+    <td align="center">Instance Segmentation</td>
+    <td align="center">COCO2017</td>
+    <td align="center">mask AP</td>
+    <td align="center">34.8</td>
+    <td align="center">-</td>
+    <td align="center">34.5</td>
+    <td align="center">-</td>
+    <td align="center">-</td>
+    <td align="center">-</td>
+    <td align="center">-</td>
+    <td align="center">-</td>
+    <td align="center">34.5</td>
+  </tr>
 </tbody>
 </table>
 </div>

docs/zh_cn/03-benchmark/supported_models.md

@@ -22,7 +22,8 @@
 | [RepPoints](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/reppoints)                       | MMDetection      |      N      |      N      |    Y     |  N   |   ?   |    Y     |   N    |  N   |
 | [DETR](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/detr)                                 | MMDetection      |      N      |      Y      |    Y     |  N   |   ?   |    N     |   N    |  N   |
 | [CenterNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/centernet)                       | MMDetection      |      N      |      Y      |    Y     |  N   |   ?   |    N     |   N    |  N   |
-| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo)                                 | MMDetection      |      N      |      N      |    N     |  N   |   N   |    Y     |   N    |  N   |
+| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo)                                 | MMDetection      |      N      |      Y      |    N     |  N   |   N   |    Y     |   N    |  N   |
+| [SOLOv2](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2)                             | MMDetection      |      N      |      Y      |    N     |  N   |   N   |    Y     |   N    |  N   |
 | [ResNet](https://github.com/open-mmlab/mmclassification/tree/1.x/configs/resnet)                        | MMClassification |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |
 | [ResNeXt](https://github.com/open-mmlab/mmclassification/tree/1.x/configs/resnext)                      | MMClassification |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |
 | [SE-ResNet](https://github.com/open-mmlab/mmclassification/tree/1.x/configs/seresnet)                   | MMClassification |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |

docs/zh_cn/04-supported-codebases/mmdet.md

@@ -192,25 +192,26 @@ cv2.imwrite('output_detection.png', img)
 
 ## 模型支持列表
 
-|                                             Model                                             |         Task          | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO |
-| :-------------------------------------------------------------------------------------------: | :-------------------: | :---------: | :------: | :--: | :---: | :------: |
-|            [ATSS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/atss)            |    ObjectDetection    |      Y      |    Y     |  N   |   N   |    Y     |
-|            [FCOS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fcos)            |    ObjectDetection    |      Y      |    Y     |  Y   |   N   |    Y     |
-|        [FoveaBox](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/foveabox)        |    ObjectDetection    |      Y      |    N     |  N   |   N   |    Y     |
-|            [FSAF](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fsaf)            |    ObjectDetection    |      Y      |    Y     |  Y   |   Y   |    Y     |
-|       [RetinaNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/retinanet)       |    ObjectDetection    |      Y      |    Y     |  Y   |   Y   |    Y     |
-|             [SSD](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/ssd)             |    ObjectDetection    |      Y      |    Y     |  Y   |   N   |    Y     |
-|           [VFNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/vfnet)           |    ObjectDetection    |      N      |    N     |  N   |   N   |    Y     |
-|           [YOLOv3](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolo)           |    ObjectDetection    |      Y      |    Y     |  Y   |   N   |    Y     |
-|           [YOLOX](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolox)           |    ObjectDetection    |      Y      |    Y     |  Y   |   N   |    Y     |
-|   [Cascade R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn)    |    ObjectDetection    |      Y      |    Y     |  N   |   Y   |    Y     |
-|    [Faster R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn)     |    ObjectDetection    |      Y      |    Y     |  Y   |   Y   |    Y     |
-| [Faster R-CNN + DCN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn)  |    ObjectDetection    |      Y      |    Y     |  Y   |   Y   |    Y     |
-|             [GFL](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/gfl)             |    ObjectDetection    |      Y      |    Y     |  N   |   ?   |    Y     |
-|       [RepPoints](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/reppoints)       |    ObjectDetection    |      N      |    Y     |  N   |   ?   |    Y     |
-|            [DETR](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/detr)            |    ObjectDetection    |      Y      |    Y     |  N   |   ?   |    Y     |
-|       [CenterNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/centernet)       |   Object Detection    |      Y      |    Y     |  N   |   ?   |    ?     |
-| [Cascade Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn) | InstanceSegmentation  |      Y      |    N     |  N   |   N   |    Y     |
-|      [Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/mask_rcnn)       | InstanceSegmentation  |      Y      |    Y     |  N   |   N   |    Y     |
-|      [Swin Transformer](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/swin)      | InstanceSegmentation  |      Y      |    Y     |  N   |   N   |    N     |
-|            [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo)            | Instance Segmentation |      N      |    N     |  N   |   N   |    Y     |
+|                                             Model                                             |         Task         | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO |
+| :-------------------------------------------------------------------------------------------: | :------------------: | :---------: | :------: | :--: | :---: | :------: |
+|            [ATSS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/atss)            |   ObjectDetection    |      Y      |    Y     |  N   |   N   |    Y     |
+|            [FCOS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fcos)            |   ObjectDetection    |      Y      |    Y     |  Y   |   N   |    Y     |
+|        [FoveaBox](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/foveabox)        |   ObjectDetection    |      Y      |    N     |  N   |   N   |    Y     |
+|            [FSAF](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fsaf)            |   ObjectDetection    |      Y      |    Y     |  Y   |   Y   |    Y     |
+|       [RetinaNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/retinanet)       |   ObjectDetection    |      Y      |    Y     |  Y   |   Y   |    Y     |
+|             [SSD](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/ssd)             |   ObjectDetection    |      Y      |    Y     |  Y   |   N   |    Y     |
+|           [VFNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/vfnet)           |   ObjectDetection    |      N      |    N     |  N   |   N   |    Y     |
+|           [YOLOv3](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolo)           |   ObjectDetection    |      Y      |    Y     |  Y   |   N   |    Y     |
+|           [YOLOX](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolox)           |   ObjectDetection    |      Y      |    Y     |  Y   |   N   |    Y     |
+|   [Cascade R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn)    |   ObjectDetection    |      Y      |    Y     |  N   |   Y   |    Y     |
+|    [Faster R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn)     |   ObjectDetection    |      Y      |    Y     |  Y   |   Y   |    Y     |
+| [Faster R-CNN + DCN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn)  |   ObjectDetection    |      Y      |    Y     |  Y   |   Y   |    Y     |
+|             [GFL](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/gfl)             |   ObjectDetection    |      Y      |    Y     |  N   |   ?   |    Y     |
+|       [RepPoints](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/reppoints)       |   ObjectDetection    |      N      |    Y     |  N   |   ?   |    Y     |
+|            [DETR](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/detr)            |   ObjectDetection    |      Y      |    Y     |  N   |   ?   |    Y     |
+|       [CenterNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/centernet)       |   Object Detection   |      Y      |    Y     |  N   |   ?   |    ?     |
+| [Cascade Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn) | InstanceSegmentation |      Y      |    N     |  N   |   N   |    Y     |
+|      [Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/mask_rcnn)       | InstanceSegmentation |      Y      |    Y     |  N   |   N   |    Y     |
+|      [Swin Transformer](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/swin)      | InstanceSegmentation |      Y      |    Y     |  N   |   N   |    N     |
+|            [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo)            | InstanceSegmentation |      Y      |    N     |  N   |   N   |    Y     |
+|          [SOLOv2](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2)          | InstanceSegmentation |      Y      |    N     |  N   |   N   |    Y     |

mmdeploy/codebase/mmdet/deploy/object_detection_model.py

@@ -44,11 +44,13 @@ def __init__(self,
                  backend_files: Sequence[str],
                  device: str,
                  deploy_cfg: Union[str, Config],
+                 model_cfg: Optional[Union[str, Config]] = None,
                  data_preprocessor: Optional[Union[dict, nn.Module]] = None,
                  **kwargs):
         super().__init__(
             deploy_cfg=deploy_cfg, data_preprocessor=data_preprocessor)
         self.deploy_cfg = deploy_cfg
+        self.model_cfg = model_cfg
         self.device = device
         self._init_wrapper(
             backend=backend, backend_files=backend_files, device=device)
@@ -195,6 +197,8 @@ def forward(self,
         img_metas = [data_sample.metainfo for data_sample in data_samples]
         results = []
         rescale = kwargs.get('rescale', True)
+        model_type = self.model_cfg.model.type if \
+            self.model_cfg is not None else None
         for i in range(batch_size):
             dets, labels = batch_dets[i], batch_labels[i]
             result = InstanceData()
@@ -234,6 +238,8 @@ def forward(self,
 
             result.scores = scores
             result.bboxes = bboxes
+            if model_type in ['SOLO', 'SOLOv2']:
+                result.bboxes = bboxes.new_zeros(bboxes.shape)
             result.labels = labels
 
             if batch_masks is not None:

mmdeploy/codebase/mmdet/models/dense_heads/__init__.py

@@ -9,5 +9,6 @@
 from . import rtmdet_head  # noqa: F401,F403
 from . import rtmdet_ins_head  # noqa: F401,F403
 from . import solo_head  # noqa: F401,F403
+from . import solov2_head  # noqa: F401,F403
 from . import yolo_head  # noqa: F401,F403
 from . import yolox_head  # noqa: F401,F403

mmdeploy/codebase/mmdet/models/dense_heads/solo_head.py

@@ -12,14 +12,15 @@
 
 
 @FUNCTION_REWRITER.register_rewriter(
-    'mmdet.models.dense_heads.SOLOHead.predict_by_feat', backend='openvino')
-def solohead__predict_by_feat__openvino(self,
-                                        mlvl_mask_preds: List[Tensor],
-                                        mlvl_cls_scores: List[Tensor],
-                                        batch_img_metas: List[Dict],
-                                        cfg: OptConfigType = None,
-                                        **kwargs):
-    """Rewrite `predict_by_feat` of `SOLOHead` for openvino backend."""
+    func_name='mmdet.models.dense_heads.solo_head.'
+    'SOLOHead.predict_by_feat')
+def solohead__predict_by_feat(self,
+                              mlvl_mask_preds: List[Tensor],
+                              mlvl_cls_scores: List[Tensor],
+                              batch_img_metas: List[Dict],
+                              cfg: OptConfigType = None,
+                              **kwargs):
+    """Rewrite `predict_by_feat` of `SOLOHead` for default backend."""
 
     ctx = FUNCTION_REWRITER.get_context()
     batch_size = mlvl_cls_scores[0].size(0)