agnostic-regression for bbox

[zimenglan-sysu-512]

No description provided.

[facebook-github-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need the corporate CLA signed.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

[zimenglan-sysu-512]

relative to #111

[fmassa]

This is awesome, thanks a lot!

I started writing some comments, but then I realized that I needed to double think a few things. I'll try to get to reviewing it fully by the end of the week.

Also, did you launched trainings with class-agnostic box regression (to compare against Detectron)?

[zimenglan-sysu-512]

hi @fmassa
i run e2e_mask_rcnn_R_50_FPN_1x with class-agnostic regression and ohem( but set POSITIVE_FRACTION to be 0.25) which i forgot to turn off (sorry for that), and get the results:

bbox
Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.380
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.593
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.414
Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.214
Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.410
Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.501
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.313
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.492
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.516
Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.324
Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.550
Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.661

mask
Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.343
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.558
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.363
Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.151
Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.367
Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.510
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.294
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.447
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.467
Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.264
Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.502
Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.626

it seems ohem helps a little. (i also train the same setting except class-agnostic regression, it got 37.9/34.4). i think the class-agnostic regression works

[fmassa]

This looks good to me, thanks a lot!

I have only one question, but I think this can be merged already.
I'd just like to discuss a bit about it before merging.

[fmassa]

this is how Detectron implements it: by using 0 for background and 1 for the other classes.

But I'm starting to wonder if this is actually necessary here: given that we index out all negative examples via sampled_pos_inds_subset, we don't backprop through the background elements anyway.
Given that, I think we could have only num_bbox_reg_classes = 1, and the rest of the code should work as well.

@rbgirshick what do you think? Am I missing something here?

[fmassa]

After a second thought, we still use the +1 for classes for the box regression, so in order to maintain consistency I'll keep the same here.

Thanks a lot!

[zimenglan-sysu-512]

hi @fmassa
after merging from master branch, i re-run the experiment e2e_mask_rcnn_R_50_FPN_1x without ohem, and get 37.6&34.1.

[fmassa]

Hi @zimenglan-sysu-512 ,

Thanks for the information, those numbers look good I think, thanks!

[xxradon]

@zimenglan-sysu-512 I noticed you had implement OHEM on maskrcnn-benchmark,can you share your code or give us some suggestions to implement.Thank you very much.

[zimenglan-sysu-512]

hi @xxradon
here is my implementation in these lines:

        # https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#cross_entropy
        reduction = "mean" # default for 'F.cross_entropy'
        if self.reduction == "none":
            reduction = self.reduction
        classification_loss = F.cross_entropy(
            class_logits, labels, reduction=reduction
        )

        # get indices that correspond to the regression targets for
        # the corresponding ground truth labels, to be used with
        # advanced indexing
        sampled_pos_inds_subset = torch.nonzero(labels > 0).squeeze(1)
        labels_pos = labels[sampled_pos_inds_subset]
        if self.cls_agnostic_bbox_reg:
            map_inds = torch.tensor([4, 5, 6, 7], device=device)
        else:
            map_inds = 4 * labels_pos[:, None] + torch.tensor(
                [0, 1, 2, 3], device=device)

        reduction = self.reduction
        box_loss = smooth_l1_loss(
            box_regression[sampled_pos_inds_subset[:, None], map_inds],
            regression_targets[sampled_pos_inds_subset],
            reduction=reduction,
            beta=1,
        )
        if self.reduction in ['sum',]:
            box_loss = box_loss / labels.numel()

        if self.reduction == 'none':
            num_props = classification_loss.size()[0]
            box_loss_paste = torch.zeros((num_props,), device=device)
            box_loss_paste[sampled_pos_inds_subset] = box_loss
            box_loss = box_loss_paste

        return classification_loss, box_loss

u should modify the corresponding smooth_l1_loss to return the losses of all rois.

and then in box_head.py to sum the loss_classifier and loss_box_reg like this:

def ohem_for_bbox_loss(ohem_batch_size, loss_classifier, loss_box_reg):
    '''select the top-K highest-loss proposals'''
    ohem_batch_size = min(
        ohem_batch_size, loss_classifier.size()[0]
    )
    cls_box_loss = loss_classifier.detach() + loss_box_reg.detach()
    ohem_idxs = torch.topk(cls_box_loss, ohem_batch_size)[1]
    loss_classifier = loss_classifier[ohem_idxs]
    loss_box_reg = loss_box_reg[ohem_idxs]
    loss_classifier = loss_classifier.mean()
    loss_box_reg = loss_box_reg.mean()
    return loss_classifier, loss_box_reg

hope it can help u.
good luck.

[zimenglan-sysu-512]

related to #439 and #630

[xxradon]

@zimenglan-sysu-512 Thank you very much,it's very helpful.