Exploit CAM by itself: Complementary Learning System for Weakly Supervised Semantic Segmentation

This repository contains the source codes for TMLR'24 paper:

Exploit CAM by itself: Complementary Learning System for Weakly Supervised Semantic Segmentation.

Author List: Wankou Yang, Jiren Mai, Fei Zhang, Tongliang Liu, Bo Han.

Introduction

Weakly Supervised Semantic Segmentation (WSSS) with image-level labels has long been suffering from fragmentary object regions led by Class Activation Map (CAM), which is incapable of generating fine-grained masks for semantic segmentation. To guide CAM to find more non-discriminating object patterns, this paper turns to an interesting working mechanism in agent learning named Complementary Learning System (CLS). CLS holds that the neocortex builds a sensation of general knowledge, while the hippocampus specially learns specific details, completing the learned patterns. Motivated by this simple but effective learning pattern, we propose a General-Specific Learning Mechanism (GSLM) to explicitly drive a coarse-grained CAM to a fine-grained pseudo mask. Specifically, GSLM develops a General Learning Module (GLM) and a Specific Learning Module (SLM). The GLM is trained with image-level supervision to extract coarse and general localization representations from CAM. Based on the general knowledge in the GLM, the SLM progressively exploits the specific spatial knowledge from the localization representations, expanding the CAM in an explicit way. To this end, we propose the Seed Reactivation to help SLM reactivate non-discriminating regions by setting a boundary for activation values, which successively identifies more regions of CAM. Without extra refinement processes, our method is able to achieve improvements for CAM of over 20.0% mIoU on PASCAL VOC 2012 and 10.0% mIoU on MS COCO 2014 datasets, representing a new state-of-the-art among existing WSSS methods.

Acknowledgement

The repository is built mainly upon these repositories:

• jiwoon-ahn/irn [1];

[1] Jiwoon Ahn et al. Weakly supervised learning of instance segmentation with inter-pixel relations, CVPR 2019.

Citation

@article{
yang2024exploit,
title={Exploit {CAM} by itself: Complementary Learning System for Weakly Supervised Semantic Segmentation},
author={Wankou Yang and Jiren Mai and Fei Zhang and Tongliang Liu and Bo Han},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2024},
url={https://openreview.net/forum?id=KutEe24Yai},
note={}
}