Using the ViT for Single-channel Images

[ahmed1996said]

Hello all,

I'm wondering how to use a pretrained ViT model to finetune on single channel images?
I tried adjusting the architecture (for vit_base_patch16_224) so that the patch embedding projection layer is

Conv2D(1,256,kernel_size=(16,16), stride=(16,16))  # D = 16*16*1 = 256

instead of

Conv2D(3,768,kernel_size=(16,16), stride=(16,16))

Similarly, I adjusted the encoder inputs/outputs to be 256 instead of 768 and made similar changes to the the attention qkv, attention projection layer and the MLP.
However, I face an error when training complaining about the sizes: Sizes of tensors must match except in dimension 2. Got 256 and 768 (The offending index is 0) triggered by the forward pass function.

Any suggestions on how to correctly use the model for single channel images?

[Abhishek-Prajapat]

Hi there.
This might be a late reply but you can use one thing and that is to stack you images. I was doing a similar problem and what I did was this:-

image = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)
image = cv2.merge((image, image, image))

[rwightman]

@Abhishek-Prajapat just create the model with arg in_chans=1 https://fastai.github.io/timmdocs/models#My-dataset-doesn't-consist-of-3-channel-images---what-now?

EDIT: also @ahmed1996said

[ahmed1996said]

Great thank you, this worked fine!

[yipliu]

How about ViTImageProcessor ?