【Bug】BertLayer should be used as a decoder model if cross attention is added

[rexainn]

Thanks for your great work! But this error stucks me for a week......Please tell me how to fix it!

Command：python inference_ram.py --image images/demo/demo1.jpg --pretrained ../ram_swin_large_14m.pth

Log:

Traceback (most recent call last):
  File "/DATA/DATA1/renxiaoyu/202405_asd/Grounded-Segment-Anything/recognize-anything/inference_ram.py", line 43, in <module>
    model = ram(pretrained=args.pretrained,
  File "/DATA/DATA1/renxiaoyu/202405_asd/Grounded-Segment-Anything/recognize-anything/ram/models/ram.py", line 399, in ram
    model = RAM(**kwargs)
  File "/DATA/DATA1/renxiaoyu/202405_asd/Grounded-Segment-Anything/recognize-anything/ram/models/ram.py", line 143, in __init__
    self.tag_encoder = BertModel(config=encoder_config,
  File "/home/huiyu/.conda/envs/intern_clean/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 891, in __init__
    self.encoder = BertEncoder(config)
  File "/home/huiyu/.conda/envs/intern_clean/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 559, in __init__
    self.layer = nn.ModuleList([BertLayer(config) for _ in range(config.num_hidden_layers)])
  File "/home/huiyu/.conda/envs/intern_clean/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 559, in <listcomp>
    self.layer = nn.ModuleList([BertLayer(config) for _ in range(config.num_hidden_layers)])
  File "/home/huiyu/.conda/envs/intern_clean/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 479, in __init__
    raise ValueError(f"{self} should be used as a decoder model if cross attention is added")
ValueError: BertLayer(
  (attention): BertAttention(
    (self): BertSelfAttention(
      (query): Linear(in_features=768, out_features=768, bias=True)
      (key): Linear(in_features=768, out_features=768, bias=True)
      (value): Linear(in_features=768, out_features=768, bias=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (output): BertSelfOutput(
      (dense): Linear(in_features=768, out_features=768, bias=True)
      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
  )
) should be used as a decoder model if cross attention is added