Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts

This is the source code for the paper: Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts (ICML 2024).

Requirements

• Python 3.8
• PyTorch 2.0
• Torchvision 0.15
• Tensorboard

• Other dependencies are listed in requirements.txt.

To install requirements, run:

conda create -n lift python=3.8 -y
conda activate lift
conda install pytorch==2.0.0 torchvision==0.15.0 pytorch-cuda=11.7 -c pytorch -c nvidia
conda install tensorboard
pip install -r requirements.txt

We encourage installing the latest dependencies. If there are any incompatibilities, please install the dependencies with the following versions.

numpy==1.24.3
scipy==1.10.1
scikit-learn==1.2.1
yacs==0.1.8
tqdm==4.64.1
ftfy==6.1.1
regex==2022.7.9
timm==0.6.12

Hardware

Most experiments can be reproduced using a single GPU with 20GB of memory (larger models such as ViT-L require more memory).

• To further reduce the GPU memory cost, gradient accumulation is recommended. Please refer to Usage for detailed instructions.

Quick Start on the CIFAR-100-LT dataset

# run LIFT on CIFAR-100-LT (with imbalanced ratio=100)
python main.py -d cifar100_ir100 -m clip_vit_b16 adaptformer True

By running the above command, you can automatically download the CIFAR-100 dataset and run the method (LIFT).

Running on Large-scale Long-tailed Datasets

Prepare the Dataset

Download the dataset Places, ImageNet, and iNaturalist 2018.

Put files in the following locations and change the path in the data configure files in configs/data:

• Places

Path/To/Dataset
├─ train
│  ├─ airfield
|  |  ├─ 00000001.jpg
|  |  └─ ......
│  └─ ......
└─ val
   ├─ airfield
   |  ├─ Places365_val_00000435.jpg
   |  └─ ......
   └─ ......

• ImageNet

Path/To/Dataset
├─ train
│  ├─ n01440764
|  |  ├─ n01440764_18.JPEG
|  |  └─ ......
│  └─ ......
└─ val
   ├─ n01440764
   |  ├─ ILSVRC2012_val_00000293.JPEG
   |  └─ ......
   └─ ......

• iNaturalist 2018

Path/To/Dataset
└─ train_val2018
   ├─ Actinopterygii
   |  ├─ 2229
   |  |  ├─ 2c5596da5091695e44b5604c2a53c477.jpg
   |  |  └─ ......
   |  └─ ......
   └─ ......

Reproduction

To reproduce the main result in the paper, please run

# run LIFT on ImageNet-LT
python main.py -d imagenet_lt -m clip_vit_b16 adaptformer True

# run LIFT on Places-LT
python main.py -d places_lt -m clip_vit_b16 adaptformer True

# run LIFT on iNaturalist 2018
python main.py -d inat2018 -m clip_vit_b16_peft adaptformer True num_epochs 20

For other experiments, please refer to scripts for reproduction commands.

Detailed Usage

To train and test the proposed method on more settings, run

python main.py -d [data] -m [model] [options]

The [data] can be the name of a .yaml file in configs/data, including imagenet_lt, places_lt, inat2018, cifar100_ir100, cifar100_ir50, cifar100_ir10, etc.

The [model] can be the name of a .yaml file in configs/model, including clip_rn50, clip_vit_b16, in21k_vit_b16, etc.

Note that using only -d and -m options denotes only fine-tuning the classifier. Please use additional [options] for more settings.

• To apply lightweight fine-tuning methods, add options like lora True, adaptformer True, etc.

• To apply test-time ensembling, add tte True.

Moreover, [options] can facilitate modifying the configure options in utils/config.py. Following are some examples.

• To specify the root path of datasets, add root Path/To/Datasets.

• To change the output directory, add an option like output_dir NewExpDir. Then the results will be saved in output/NewExpDir.

• To assign a single GPU (for example, GPU 0), add an option like gpu 0.

• To apply gradient accumulation, add micro_batch_size XX. This can further reduce GPU memory costs. Note that XX should be a divisor of batch_size.

• To test an existing model, add test_only True. This option will test the model trained by your configure file. To test another model, add an additional option like model_dir output/AnotherExpDir.

• To test an existing model on the training set, add test_train True.

You can also refer to scripts for example commands.

Acknowledgment

We thank the authors for the following repositories for code reference: [OLTR], [Classifier-Balancing], [Dassl], [CoOp].

Citation

If you find this repo useful for your work, please cite as:

@inproceedings{shi2024longtail,
  title={Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts},
  author={Jiang-Xin Shi and Tong Wei and Zhi Zhou and Jie-Jing Shao and Xin-Yan Han and Yu-Feng Li},
  booktitle={Proceedings of the 41st International Conference on Machine Learning},
  year={2024}
}