Panoptic Segmentation

[karanchahal]

Hello,

This pull request adds a demo which modifies a Mask RCNN to construct a Panoptic Feature Pyramid Network. The google colab notebook linke is given here. The notebook details an example which trains a model on the shapes dataset that outputs:

1. Bounding boxes and instance segmentation masks via the Mask RCNN
2. Semantic segmentation masks through the Panoptic Segmentation Head which differentiates the background and foreground.

I hope to build on this demo and add full fledged Panoptic Segmentation support ( #337 ) to the repo, this demo is meant to be as a proof of concept (or a sanity check) that things work.

The basic stuff is here, however we will need to add support for Panoptic Evaluation and modify the structures directory to support the panoptic format. I would love to discuss on what would be the best way to add this support and perform ablation studies on verifying the accuracy of the model.

Would love to know what you think @fmassa !

Thanks,
Karan

[fmassa]

I think this is a great idea!

I'm glad to merge this as is.

Also, it might be good to connect with the authors of https://arxiv.org/abs/1906.05896
They have extended maskrcnn-benchmark to support Panoptic Segmentation, I was wondering if they were planning on open-sourcing their implementation?

cc @jlazarow , the main author of https://arxiv.org/abs/1906.05896

[karanchahal]

@jlazarow Your thoughts would be much appreciated. What do you think would be the best way to go about including support for Panoptic Segmentation in this repo ?

[fmassa]

Merging this in the meantime, as it's self-contained and looks great!

[jlazarow]

Hi,

Yes, we will open source the implementation in the near future. I can write some notes/thoughts this week and can upload our rough implementations of the Panoptic FPN/fusion code. I do not expect these to be immediately merge-able (I might have done narrow-minded things in coco_eval.py), but I would be happy to help get it into this repo (I guess before Detectron2 drops).

[karanchahal]

Yeah, that would be great ! I'm looking forward to it.

On a side note, what's detectron 2 going to support ? Sounds very interesting.

[karanchahal]

Hello @jlazarow !
Is there any update to this ? I would love to get started on this as per your instructions :D