Long-Tailed-Classification-Leaderboard

date: 2021/3/3（Updated 2021/3/3）
auther: YW YSZ

1. Introduction

List of abbreviations:

Abbreviations	ReW	TrL	MeL	DeL	Aug	SeSu	OtM
Full names	Re-weighting	Transfer Learning	Meta Learning	Decoupling Learning	Data Augmentation	Self-Supervised Learning	Other methods

2. Benchmark datasets

Dataset	Year	Images(Triain/Val/Test)	Classes	Max images	Min images	Imbalance factor	Reported by
CIFAR-LT-10	2019	50000–11203/--/10000	10	5,000	500–25	1.0–200.0	Source
CIFAR-LT-100	2019	50000–9502/--/10000	100	500	500–2	1.0–200.0	Source
ImageNet-LT	2019	115846/20000/50000	1000	1280	5	256.0	Source
Places-LT	2019	62500/7300/36500	365	4980	5	996.0	Source
iNat 2017	2017	579184/95986/--	5089	3919	9	435.4	Source
iNat 2018	2018	437513/24426/--	8142	1000	2	500.0	Source

3. Leaderboard

3.1 CIFAR-10-LT

Evaluation metric: classification error rate.

IF represents Imbalance factor.

Backbone: ResNet-32

Method	Venue	Year	Backbone	Type	IF=10	IF=50	IF=100	Code	Reported by
Class-Balanced Loss	CVPR	2019	ResNet-32	ReW	12.51	20.73	25.43	----	Source
LDAM-DRW	NeurIPS	2019	ResNet-32	ReW	11.84	-----	22.97	----	Source
MW-Net	NeurIPS	2019	ResNet-32	ReW/MeL	12.16	19.94	24.79	----	Source
LDAM-M2m	CVPR	2020	ResNet-32	TrL	12.5	-----	20.9	-----	Source
BBN	CVPR	2020	ResNet-32	DeL	11.68	17.82	20.18	----	Source
CBasDA-LDAM	CVPR	2020	ResNet-32	ReW	12.6	17.77	22.77	----	Source
Balanced Softmax	ECCV-Workshop	2020	ResNet-32	OtM	9.1	-----	16.9	----	Source
De-confound-TDE	NeurIPS	2020	ResNet-32	OtM	11.5	16.4	19.4	----	Source
BALMS	NeurIPS	2020	ResNet-32	ReW	8.7	-----	15.1	----	Source
LDAM-DRW + SSP	NeurIPS	2020	ResNet-32	SeSu	11.47	17.87	22.17	----	Source
Baseline + tricks	AAAI	2021	ResNet-32	OtM	-----	16.41	19.97	----	Source
Remix-DRW	ECCV-Workshop	2020	ResNet-32	Aug	10.98	-----	20.24	----	Source

Backbone: ResNet-18

Method	Venue	Year	Backbone	Type	IF=10	IF=50	IF=100	Code	Reported by
FSA	ECCV	2020	ResNet-18	Aug	8.25	15.29	19.43	----	Source

Backbone: ResNet-34

Method	Venue	Year	Backbone	Type	IF=10	IF=50	IF=100	Code	Reported by
FSA	ECCV	2020	ResNet-34	Aug	8.8	15.51	17.94	----	Source

3.2 CIFAR-100- LT

Evaluation metric: classification error rate.

Backbone: ResNet-32

Method	Venue	Year	Backbone	Type	IF=10	IF=50	IF=100	Code	Reported by
Class-Balanced Loss	CVPR	2019	ResNet-32	ReW	42.01	54.68	60.40	----	Source
LDAM-DRW	NeurIPS	2019	ResNet-32	ReW	41.29	-----	57.96	----	Source
MW-Net	NeurIPS	2019	ResNet-32	ReW/MeL	41.54	53.26	57.91	----	Source
LDAM-M2m	CVPR	2020	ResNet-32	TrL	42.4	-----	56.5	-----	Source
BBN	CVPR	2020	ResNet-32	DeL	40.88	52.98	57.44	----	Source
CBasDA-LDAM	CVPR	2020	ResNet-32	ReW	42.0	50.84	60.47	----	Source
LFME+LDAM	ECCV	2020	ResNet-32	TrL	-----	-----	56.2	----	Source
Balanced Softmax	ECCV-Workshop	2020	ResNet-32	OtM	36.9	-----	49.7	----	Source
De-confound-TDE	NeurIPS	2020	ResNet-32	OtM	40.4	49.7	55.9	----	Source
BALMS	NeurIPS	2020	ResNet-32	ReW	37.0	-----	49.2	----	Source
LDAM-DRW + SSP	NeurIPS	2020	ResNet-32	SeSu	41.09	52.89	56.57	----	Source
Baseline + tricks	AAAI	2021	ResNet-32	OtM	-----	48.31	52.17	----	Source
Remix-DRW	ECCV-Workshop	2020	ResNet-32	Aug	38.77	-----	53.23	----	Source

Backbone: ResNet-18

Method	Venue	Year	Backbone	Type	IF=10	IF=50	IF=100	Code	Reported by
FSA	ECCV	2020	ResNet-18	Aug	34.92	48.1	53.43	----	Source

Backbone: ResNet-34

Method	Venue	Year	Backbone	Type	IF=10	IF=50	IF=100	Code	Reported by
FSA	ECCV	2020	ResNet-34	Aug	34.71	47.83	51.49	----	Source

3.3 ImageNet-LT

Evaluation metric: closed-set setting/Top-1 classification accuracy.

Backbone: ResNet-10

Method	Venue	Year	Backbone	Type	Many-Shot	Medium-Shot	Few-Shot	ALL	Code	Reported by
OLTR	CVPR	2019	ResNet-10	TrL	43.2	35.1	18.5	35.6	----	Source
LWS	ICLR	2020	ResNet-10	DeL	-----	-----	----	41.4	----	Source
IEM	CVPR	2020	ResNet-10	OtM	48.9	44.0	24.4	43.2	----	Source
LFME+OLTR	ECCV	2020	ResNet-10	TrL	47.0	37.9	19.2	38.8	----	Source
FSA	ECCV	2020	ResNet-10	Aug	47.3	31.6	14.7	35.2	----	Source
BALMS	NeurIPS	2020	ResNet-10	ReW	50.3	39.5	25.3	41.8	----	Source
cRT + SSP	NeurIPS	2020	ResNet-10	SeSu	-----	-----	----	43.2	----	Source
Baseline + tricks	AAAI	2021	ResNet-10	OtM	-----	-----	----	43.31	----	Source

Backbone: ResNeXt-50

Method	Venue	Year	Backbone	Type	Many-Shot	Medium-Shot	Few-Shot	ALL	Code	Reported by
LWS	ICLR	2020	ResNeXt-50	DeL	60.2	47.2	30.3	49.9	----	Source

Backbone: ResNeXt-152

Method	Venue	Year	Backbone	Type	Many-Shot	Medium-Shot	Few-Shot	ALL	Code	Reported by
LWS	ICLR	2020	ResNeXt-152	DeL	63.5	50.4	34.2	53.3	----	Source

3.4 Places-LT

Evaluation metric: closed-set setting/Top-1 classification accuracy.

Backbone: ResNet-152

Method	Venue	Year	Backbone	Type	Many-Shot	Medium-Shot	Few-Shot	ALL	Code	Reported by
OLTR	CVPR	2019	ResNet-152	TrL	44.7	37	25.3	35.9	----	Source
LWS	ICLR	2020	ResNet-152	DeL	40.6	39.1	28.6	37.6	----	Source
τ -normalized	ICLR	2020	ResNet-152	DeL	37.8	40.7	31.8	37.9	----	Source
IEM	CVPR	2020	ResNet-152	OtM	46.8	39.2	28.0	39.7	----	Source
LFME+OLTR	ECCV	2020	ResNet-152	TrL	39.3	39.6	24.2	36.2	----	Source
FSA	ECCV	2020	ResNet-152	Aug	42.8	37.5	22.7	36.4	----	Source

Backbone: ResNet-10

Method	Venue	Year	Backbone	Type	Many-Shot	Medium-Shot	Few-Shot	ALL	Code	Reported by
BALMS	NeurIPS	2020	ResNet-10	ReW	41.2	39.8	31.6	38.7	----

3.5 iNaturalist

Evaluation metric: Top-1 classification accuracy

Backbone: ResNet-50

Method	Venue	Year	Backbone	Type	iNat-2017(Top1)	iNat-2018(Top1)	Code	Reported by
CB Focal	CVPR	2019	ResNet-50	ReW	58.08	61.12	----	Source
LWS	ICLR	2020	ResNet-50	DeL	-----	65.9/69.5 (90/200)	----	Source
IEM	CVPR	2020	ResNet-50	OtM	-----	70.2	----	Source
BBN	CVPR	2020	ResNet-50	DeL	63.39	66.29	----	Source
BBN(2×)	CVPR	2020	ResNet-50	DeL	65.75	69.62	----	Source
CBasDA-CE	CVPR	2020	ResNet-50	ReW	59.38	67.55	----	Source
FSA	ECCV	2020	ResNet-50	Aug	61.96	65.91	----	Source
cRT + SSP	NeurIPS	2020	ResNet-50	SeSu	-----	68.1	----	Source
Baseline + tricks	AAAI	2021	ResNet-50	OtM	-----	70.87	----	Source
Remix-DRS	ECCV-Workshop	2020	ResNet-50	Aug	-----	70.74	----	Source

Backbone: ResNet-152

Method	Venue	Year	Backbone	Type	iNat-2017(Top1)	iNat-2018(Top1)	Code	Reported by
CB Focal	CVPR	2019	ResNet-152	ReW	61.84	64.16	----	Source
LWS	ICLR	2020	ResNet-152	DeL	-----	69.1/72.1 (90/200)	----	Source
FSA	ECCV	2020	ResNet-152	Aug	66.58	69.08	----	Source

4. Contact

Yan Wang : yanwang@smail.nju.edu.cn

Yongshun Zhang: zhangys@lamda.nju.edu.cn

5. References

2019

• Shu et.al., Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting, NeurIPS 2019.
• Cao et.al., Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, NeurIPS 2019.
• Cui et.al., Class-Balanced Loss Based on Effective Number of Samples, CVPR 2019.

2020

• Tang et.al., Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect, NeurIPS 2020.
• Yang et.al., Rethinking the Value of Labels for Improving Class-Imbalanced Learning, NeurIPS 2020.
• Ren et.al., Balanced Meta-Softmax for Long-Tailed Visual Recognition, NeurIPS 2020.
• Kang et.al., Decoupling Representation and Classifier for Long-Tailed Recognition, ICLR 2020.
• Kim et.al., M2m: Imbalanced Classification via Major-to-minor Translation, CVPR 2020.
• Zhou et.al., BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition, CVPR 2020.
• Jamal et.al., Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective, CVPR 2020.
• Zhu et.al., Inflated Episodic Memory with Region Self-Attention for Long-Tailed Visual Recognition, CVPR 2020.
• Liu et.al., Large-Scale Long-Tailed Recognition in an Open World, CVPR 2019.
• Xiang et.al., Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification, ECCV 2020.
• Chu et.al., Feature Space Augmentation for Long-Tailed Data, ECCV 2020.
• Ren et.al., Balanced Activation for Long-tailed Visual Recognition, ECCV 2020.
• Chou et.al., Remix: Rebalanced Mixup, ECCV'2020 Workshop.

2021

• Zhang et.al., Bag of Tricks for Long-Tailed Visual Recognition with Deep Convolutional Neural Networks, AAAI 2021.