Adversarial attacks for YOLOv8 Detection task

[BilAlHomsi]

Hi,

I'm interested in attacking yolov8 to test methods for better resistance.

The yolo model is not accepted in torchattacks. Instead, I tried to import and attack a pre-implemented model from this repo and use its adversarial samples for yolo, but it didn't work out well. I believe it is because these models work as classifiers and yolo works as a detector, among other things.

There are other previously implemented models for object detection, e.g. FRCNN, SSD., when I pass like in the attack, I get the following error message:

TypeError: cross_entropy_loss(): argument 'input' (position 1) must be Tensor, not list

It is because the object detection models do not return a tensor, but a list of tensors, where the tensors contain the boxes, the labels and the scores.

I ask for help to solve this problem

Thanks a lot!

[rikonaka]

Hi @BilAlHomsi , let's start by stating that I'm not an expert in the field of target detection, and secondly, based on your description

I believe it is because these models work as classifiers and yolo works as a detector, among other things.

this seems like a very daunting task. I just spent a little time looking into the basics of yolo and here are some of my suggestions.

If you can crop the target part of the image by yolo, resize it to a suitable size, then input it into another CNN classifier, for attack, after that return the result of the attack and use it cover original image specific region, it will be less work than modifying the whole library (again I'm not a yolo expert so if you have a better solution that's fine 😘).

I just cursory reading this paper (https://arxiv.org/abs/2202.04781) and realized that if you want to attack yolo using an attack on CNNs (such as the technique used by torchattacks) it's not possible. Attacks on yolo and attacks on CNNs belong to two field.

[wntic]

Hi @BilAlHomsi !

I also tried to apply torchattacks to the YOLOv8 model from ultralytics.

The thing is that YOLOv8 does not return torch.Tensor, but a list. This list contains a lot of data, starting from the original image and ending with masks, boxes etc. You can see it here: https://docs.ultralytics.com/modes/predict/#working-with-results

Therefore, to perform attacks using torchattacks, you need to write a class with a YOLO model, in which you need to override the forward method so that it returns exactly the segmentation masks.

from ultralytics import YOLO
import torch


class YOLOv8:
    def __init__(self):
        self.model = YOLO("yolov8l-seg.pt")
        
    def forward(self, x):
        result = self.model.predict(source=x, save=True, conf=0.5)
        masks = result[0].masks.data
        final_mask = torch.zeros_like(masks[0], dtype=torch.int)
        for i, mask in enumerate(masks):
            final_mask[mask.bool()] = i + 1
        return final_mask

Now you can use the model as model(image_tensor) and get segmentation masks as output.

Another problem, as far as I understand, is that torchattacks puts the model into evaluation mode by calling model.eval(). In YOLOv8, calling this method puts the model into training mode.

These are my observations and the problems I encountered. If you found a solution to this problem and were able to use torchattacks, please provide your solution here!