[ ReadMe: Object Detection Experiments ]

[IemProg]

Hi @bjzhb666 ,

Thanks a lot for releasing the code.

I have been closely studying your implementation and have a couple of questions that I hope you can help clarify:

1. Difference between train_own_forget_cl.py and train_own_forget.py:
   I noticed that there are two seemingly similar scripts in the repository: train_own_forget_cl.py and train_own_forget.py. Could you please elaborate on the specific differences between these two files? It would be helpful to understand their distinct purposes and when each script should be used.

2. Reproducing Object Recognition Results with DETR:
   I am particularly interested in reproducing the object recognition results you achieved using DETR. Could you provide more detailed instructions or a guide on how to set up and execute the code for this task?

3. Loss function:
   Why do you freeze the loss function parameters ?

Thank you once again for your impressive work and for any assistance you can provide.

[bjzhb666]

Thanks for your attention to our work.

1. They are essentially no different. train_own_forget.py is for single-step forgetting and train_own_forget_cl.py is for continual forgetting. But you can use train_own_forget_cl.py to conduct single-step forgetting. We use train_own_forget.py to conduct more ablation studies in single-step forgetting. If you do not want to see the details of the ablation study, just use train_own_forget_cl.py.
2. We did not release the code for Object Detection. (actually, we have not cleaned it) The procedure is similar. We use the official checkpoint provided by deformable DETR. We need to use the LoRA module to modify the network structure and add our GS loss, Knowledge Retention Loss, and Selective Forgetting Loss.
3. Actually, the loss function parameters are the FFN module (classifier). The name comes from our code base Face Transformer and is somewhat confusing. Please refer to our paper for more details (e.g. Sec.6.1) about why we freeze the FFN layers.

[NilakshanKunananthaseelan]

Hi, this is interesting work. How did you create the forget and retain datasets for each class? COCO images come with multiple instances of objects; how did you make the split to forget a particular category

[bjzhb666]

@NilakshanKunananthaseelan Thanks for your interest, that is an excellent question.

Here is an example of how we construct the dataset.

Let's say we have a cat and a dog in our picture.
If we want to forget/remain the cat and the dog at the same time, then we annotate the cat and the dog in the forgotten/remaining dataset.
If we want to forget the dog and keep the cat, then this picture can appear both in the forgotten dataset and the remaining dataset, but the annotations are different. In the forgotten dataset, we just annotate the dog, in the remaining dataset, we just annotate the cat.

[jedapaw]

The link you provided for pre-trained Face Transformer is not working; it says "Access Denied."
{ https://drive.google.com/file/d/1Qqeb8fx2_kLeGA1uuLzutpauvyqY3CSV/view?usp=drive_link }

I would like to ask:

How can I create a file structure like the following for ImageNet?

├── faces_webface_112x112_sub100_train_test
│   ├── test
│   └── train
├── imagenet100
│   ├── imagenet_classes.txt
│   ├── test
│   └── train
└── Labels.json

I am unable to find the test and train folders for ImageNet, as well as the imagenet_classes.txt file, in the 17GB Kaggle dataset.

[bjzhb666]

Thanks for your question. We set the wrong access permission. Sorry about that, we have already changed it. You can access to it now.

For the imagenet100 dataset, we should change the name of the folders in the picture below. Change 4 train.X1/2/3/4 folders to train and the val.X folder to test.

For the imagenet_classes.txt, I think we can use this https://github.com/pytorch/hub/blob/master/imagenet_classes.txt. If it works, please tell me. If you have other questions, feel free to leave a message.

Thanks again for your questions!

[jedapaw]

Thank you for updating the access permissions. I can access the link now.

Regarding the ImageNet100 dataset, I will rename the following folders accordingly:

• train.X1, train.X2, train.X3, train.X4 → train
• val.X → test

So, can I rename solely one of the datasets as train, instead of renaming all of them?

For imagenet_classes.txt, I will try using the file from this GitHub link. If it works as expected, I will let you know.

I appreciate your prompt response!

I have another question:

Do I need to create a folder named ViT-P8S8_casia100_cosface_s1-1200-150de-depth6new?

Should I place the downloaded ViT-P8S8_casia100_cosface_s1-1200-150de-depth6new inside it as follows?

└── ViT-P8S8_casia100_cosface_s1-1200-150de-depth6new
    ├── Backbone_VIT_Epoch_1185_Batch_45020_Time_2024-09-26-03-26_checkpoint.pth
    └── config.txt

Where can I find config.txt?

[NilakshanKunananthaseelan]

@NilakshanKunananthaseelan Thanks for your interest, that is an excellent question.

Here is an example of how we construct the dataset.

Let's say we have a cat and a dog in our picture. If we want to forget/remain the cat and the dog at the same time, then we annotate the cat and the dog in the forgotten/remaining dataset. If we want to forget the dog and keep the cat, then this picture can appear both in the forgotten dataset and the remaining dataset, but the annotations are different. In the forgotten dataset, we just annotate the dog, in the remaining dataset, we just annotate the cat.

Thanks for your reply.
If I want to forget dog class, then I'll only have annotations for Dog. How did you handle the cases where a fine-tuned model predicts cat classes from forgetting the dataset correctly? Did you ignore them?

[bjzhb666]

Thank you for updating the access permissions. I can access the link now.

Regarding the ImageNet100 dataset, I will rename the following folders accordingly:

• train.X1, train.X2, train.X3, train.X4 → train
• val.X → test

So, can I rename solely one of the datasets as train, instead of renaming all of them?

For imagenet_classes.txt, I will try using the file from this GitHub link. If it works as expected, I will let you know.

I appreciate your prompt response!

I have another question:

Do I need to create a folder named ViT-P8S8_casia100_cosface_s1-1200-150de-depth6new?

Should I place the downloaded ViT-P8S8_casia100_cosface_s1-1200-150de-depth6new inside it as follows?

└── ViT-P8S8_casia100_cosface_s1-1200-150de-depth6new
    ├── Backbone_VIT_Epoch_1185_Batch_45020_Time_2024-09-26-03-26_checkpoint.pth
    └── config.txt

Where can I find config.txt?

You should move all the subfolders in train.X1,2,3,4 to a new folder named train. train.X1 just part of the total training images.

You need to create a folder named ViT-P8S8_casia100_cosface_s1-1200-150de-depth6new. config.txt is just a config file when we pretrain our models, we do not need it.

[bjzhb666]

@NilakshanKunananthaseelan Thanks for your interest, that is an excellent question.
Here is an example of how we construct the dataset.
Let's say we have a cat and a dog in our picture. If we want to forget/remain the cat and the dog at the same time, then we annotate the cat and the dog in the forgotten/remaining dataset. If we want to forget the dog and keep the cat, then this picture can appear both in the forgotten dataset and the remaining dataset, but the annotations are different. In the forgotten dataset, we just annotate the dog, in the remaining dataset, we just annotate the cat.

Thanks for your reply. If I want to forget dog class, then I'll only have annotations for Dog. How did you handle the cases where a fine-tuned model predicts cat classes from forgetting the dataset correctly? Did you ignore them?

That depends. We should consider whether we want to forget the cat at the same time. Please refer to the example I mentioned before.

If you refer to the case that we want to forget the dog and keep the cat, in the forgotten dataset, we ignore the annotation for the cat in that image. You are correct.

[jedapaw]

"Code is in run_sub.sh

bash scripts/run_sub.sh

test_sub.sh is the test code for our Face Transformer. You can test the pre-trained model use it.

bash scripts/test_sub.sh"

What does this do? I tried running it, but it gave a pipeline error without the WAN database. With the WAN database, it didn't run properly I assume.

Please look below:

[jedapaw]

You should move all the subfolders in train.X1,2,3,4 to a new folder named train. train.X1 just part of the total training images.

You need to create a folder named ViT-P8S8_casia100_cosface_s1-1200-150de-depth6new. config.txt is just a config file when we pretrain our models, we do not need it.

Understood. Please consider updating the file structure in the README to ensure clarity, as I found the folder structure and training folders confusing, and others might do as well.

Thank you for your response and for the work you have done and continue to do on the repository.

[bjzhb666]

"Code is in run_sub.sh

bash scripts/run_sub.sh

test_sub.sh is the test code for our Face Transformer. You can test the pre-trained model use it.

bash scripts/test_sub.sh"

What does this do? I tried running it, but it gave a pipeline error without the WAN database. With the WAN database, it didn't run properly I assume.

Please look below:

Sorry, I do not understand your question based on these three images. run_sub.sh is used to pretrain the model. If you download it from the Google link, you do not need it.

[jedapaw]

Sorry, I do not understand your question based on these three images. run_sub.sh is used to pretrain the model. If you download it from the Google link, you do not need it.

I want to train my own model. How do I do it? Can you guide me a little?
I also want to change ViT, but I am unable to do so. A bit of guidance on how you structured everything would be greatly appreciated.

[jedapaw]

bash scripts/run_cl_forget.sh

Look at this when I ran the script:

[bjzhb666]

@jedapaw
Models are in vit_pytorch_face. The code is quite easy to follow, try it!

For your second question, I guess you are using multiple GPUs, please use a single GPU. Only the pretrain code can support multiple GPUs, other codes can only run using one GPU.