# VisualBert model producing RuntimeError

**URL:** <https://discuss.huggingface.co/t/visualbert-model-producing-runtimeerror/66360>\
**Category:** 🤗Transformers\
**Created:** [December 19, 2023, 4:58pm UTC](https://discuss.huggingface.co/t/visualbert-model-producing-runtimeerror/66360 "2023-12-19T16:58:28Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![dutta18](https://avatars.discourse-cdn.com/v4/letter/d/b487fb/32.png) [@dutta18](https://discuss.huggingface.co/u/dutta18)\
**Post date:** [December 19, 2023, 4:58pm UTC](https://discuss.huggingface.co/t/visualbert-model-producing-runtimeerror/66360/1 "2023-12-19T16:58:28Z")

</div>

I’m trying to run the VisualBERT for MCQ as given in the Huggingface model\_docs page. I generated the visual embeddings with resnet. But the model forward pass producing runtimeError. Please help.

```auto

import torch
from torch import nn
from torchvision import models, transforms
from PIL import Image as img

# Define pre-trained ResNet model and freeze convolutional layers

resnet_model = models.resnet18(pretrained=True)
for param in resnet_model.parameters():
    param.requires_grad = False

# Define transformation for image pre-processing
transform = transforms.Compose([
    transforms.Resize((224,224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

# Function to generate embedding for a single image
def get_visual_embeddings(image):
    # Preprocess image and forward through ResNet
    image = transform(image)
    image = image.unsqueeze(0) # Add batch dimension
    with torch.no_grad():
        embedding = resnet_model(image)[0] # Extract features from output

    return embedding

# Assumption: *get_visual_embeddings(image)* gets the visual embeddings of the image in the batch.
from transformers import AutoTokenizer, VisualBertForMultipleChoice
import torch

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = VisualBertForMultipleChoice.from_pretrained("uclanlp/visualbert-vcr")

prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
choice0 = "It is eaten with a fork and a knife."
choice1 = "It is eaten while held in the hand."

visual_embeds = get_visual_embeddings(image)
# (batch_size, num_choices, visual_seq_length, visual_embedding_dim)
visual_embeds = visual_embeds.expand(1, 2, *visual_embeds.shape)
visual_token_type_ids = torch.ones(visual_embeds.shape[:-1], dtype=torch.long)
visual_attention_mask = torch.ones(visual_embeds.shape[:-1], dtype=torch.float)

labels = torch.tensor(0).unsqueeze(0) # choice0 is correct (according to Wikipedia ;)), batch size 1

encoding = tokenizer([[prompt, prompt], [choice0, choice1]], return_tensors="pt", padding=True)
# batch size is 1
inputs_dict = {k: v.unsqueeze(0) for k, v in encoding.items()}
inputs_dict.update(
    {
        "visual_embeds": visual_embeds,
        "visual_attention_mask": visual_attention_mask,
        "visual_token_type_ids": visual_token_type_ids,
        "labels": labels,
    }
)

outputs = model(**inputs_dict)

```

But when I’m doing the forward pass it producing an error **“RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 2 but got size 1 for tensor number 1 in the list.”** in the torch.cat function. Where it is going wrong I couldn’t understand. Please help.

Here is the whole stack trace

```auto
RuntimeError Traceback (most recent call last)
Cell In[58], line 1
----> 1 outputs = model(**inputs_dict)

File ~/miniconda3/envs/blip_vqa_base_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499 or _global_backward_pre_hooks or _global_backward_hooks
   1500 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501 return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File ~/miniconda3/envs/blip_vqa_base_env/lib/python3.8/site-packages/transformers/models/visual_bert/modeling_visual_bert.py:1131, in VisualBertForMultipleChoice.forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, visual_embeds, visual_attention_mask, visual_token_type_ids, image_text_alignment, output_attentions, output_hidden_states, return_dict, labels)
   1120 visual_attention_mask = (
   1121 visual_attention_mask.view(-1, visual_attention_mask.size(-1))
   1122 if visual_attention_mask is not None
   1123 else None
   1124 )
   1125 visual_token_type_ids = (
   1126 visual_token_type_ids.view(-1, visual_token_type_ids.size(-1))
   1127 if visual_token_type_ids is not None
   1128 else None
   1129 )
-> 1131 outputs = self.visual_bert(
   1132 input_ids,
   1133 attention_mask=attention_mask,
   1134 token_type_ids=token_type_ids,
   1135 position_ids=position_ids,
   1136 head_mask=head_mask,
   1137 inputs_embeds=inputs_embeds,
   1138 visual_embeds=visual_embeds,
   1139 visual_attention_mask=visual_attention_mask,
   1140 visual_token_type_ids=visual_token_type_ids,
   1141 image_text_alignment=image_text_alignment,
   1142 output_attentions=output_attentions,
   1143 output_hidden_states=output_hidden_states,
   1144 return_dict=return_dict,
   1145 )
   1147 _, pooled_output = outputs[0], outputs[1]
   1149 pooled_output = self.dropout(pooled_output)

File ~/miniconda3/envs/blip_vqa_base_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499 or _global_backward_pre_hooks or _global_backward_hooks
   1500 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501 return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File ~/miniconda3/envs/blip_vqa_base_env/lib/python3.8/site-packages/transformers/models/visual_bert/modeling_visual_bert.py:796, in VisualBertModel.forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, visual_embeds, visual_attention_mask, visual_token_type_ids, image_text_alignment, output_attentions, output_hidden_states, return_dict)
    793 # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
    794 # ourselves in which case we just need to make it broadcastable to all heads.
    795 if visual_embeds is not None:
--> 796 combined_attention_mask = torch.cat((attention_mask, visual_attention_mask), dim=-1)
    797 extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(
    798 combined_attention_mask, (batch_size, input_shape + visual_input_shape)
    799 )
    801 else:

RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 2 but got size 1 for tensor number 1 in the list.

```

Please let me know where Im doing it wrong.

---

<div class="post-metadata">

**Author:** ![enochlev](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/enochlev/32/22267_2.png) [@enochlev](https://discuss.huggingface.co/u/enochlev)\
**Post date:** [December 19, 2023, 6:40pm UTC](https://discuss.huggingface.co/t/visualbert-model-producing-runtimeerror/66360/2 "2023-12-19T18:40:28Z")

</div>

its a dimension error.

this means llama was expecting a list of sentences, but you gave it only one sentence.

to be specific

```auto
Expected size 2 but got size 1

```

means you gave it a 1d array but it wanted a 2d array. In more technical sense, you provided a array of token\_ids, but it expected an array of an array of token\_ids.

To debug this I would get the shape of every item in inputs\_dict and find out which one has a shape of 1 rather then 2. Then turn it into a 2d array.

I cannot run your code (image is not defined)

---

<div class="post-metadata">

**Author:** ![dutta18](https://avatars.discourse-cdn.com/v4/letter/d/b487fb/32.png) [@dutta18](https://discuss.huggingface.co/u/dutta18)\
**Post date:** [December 22, 2023, 7:30am UTC](https://discuss.huggingface.co/t/visualbert-model-producing-runtimeerror/66360/3 "2023-12-22T07:30:42Z")

</div>

Hi, its not llama rather its VisualBERT which using BERT as a language model.

And I have used this image: [images](https://drive.google.com/file/d/1DMkdH8A965_5g4_TKna0_KqK9N3KdJp7/view?usp=sharing)

---

<div class="post-metadata">

**Author:** ![dutta18](https://avatars.discourse-cdn.com/v4/letter/d/b487fb/32.png) [@dutta18](https://discuss.huggingface.co/u/dutta18)\
**Post date:** [December 22, 2023, 7:31am UTC](https://discuss.huggingface.co/t/visualbert-model-producing-runtimeerror/66360/4 "2023-12-22T07:31:01Z")

</div>

Can you please check once where is the issue ?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [December 22, 2023, 8:04am UTC](https://discuss.huggingface.co/t/visualbert-model-producing-runtimeerror/66360/5 "2023-12-22T08:04:46Z")

</div>

Hi,

VisualBERT was a really nice model as it was one of the first to make BERT multimodal. However, there are several models available nowadays that are a lot better (VisualBERT is from 2019).

An example is [ViLT](https://huggingface.co/docs/transformers/model_doc/vilt).

What’s your use case? In case you’re only interested in having good image features, it’s recommended to take a look at more recent models like [DINOv2](https://huggingface.co/docs/transformers/main/model_doc/dinov2) and [SigLIP](https://huggingface.co/timm/ViT-B-16-SigLIP).

---

<div class="post-metadata">

**Author:** ![dutta18](https://avatars.discourse-cdn.com/v4/letter/d/b487fb/32.png) [@dutta18](https://discuss.huggingface.co/u/dutta18)\
**Post date:** [December 22, 2023, 9:36am UTC](https://discuss.huggingface.co/t/visualbert-model-producing-runtimeerror/66360/6 "2023-12-22T09:36:03Z")

</div>

Hi, thanks for asking my use case is a Visual Question Answering task based on a classification settings so I opted for VisualBERT.

And I have already working with Vilt since the last 5 months, I got an accuracy of 72%, however I need to improve on this so looking out for other VQA models. Would be glad if you could help.

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [December 22, 2023, 10:11am UTC](https://discuss.huggingface.co/t/visualbert-model-producing-runtimeerror/66360/7 "2023-12-22T10:11:15Z")

</div>

Ok, for visual question answering (VQA) it depends a bit, you have 2 kinds of models:

- models like ViLT and VisualBERT are classifiers (discriminative). They treat the problem as a multi-label classification problem: given an image, they select one or more potential classes (answers) given the question. [This fine-tuned ViLT model](https://huggingface.co/dandelin/vilt-b32-finetuned-vqa) for instance has been trained to classify between one of 3000+ possible classes. The advantage here is that these models are typically lightweight, but they have the requirement that you have a list of potential answers (classes) to the questions.
- all newer models are generative, meaning they just generate the answer in an autoregressive manner similar to ChatGPT with vision. The benefit here is that they are able to generate anything, however they are typically quite a bit larger/heavier than the discriminative models. Examples here are [BLIP-2](https://huggingface.co/docs/transformers/v4.28.1/model_doc/blip-2), [InstructBLIP](https://huggingface.co/docs/transformers/main/model_doc/instructblip), and more recently [LLaVa](https://huggingface.co/docs/transformers/main/en/model_doc/llava) and [CogVLM](https://huggingface.co/THUDM/cogvlm-chat-hf). These are multimodal extensions of large language models (LLMs). As these models are often giantic, you need an A10 or A100 GPU (or multiple) to train them. With methods like PEFT (parameter efficient fine-tuning with methods like Q-LoRa), you can fine-tune them on a single A10/A100. See [this example notebook](https://github.com/huggingface/notebooks/blob/main/peft/Fine_tune_BLIP2_on_an_image_captioning_dataset_PEFT.ipynb) which enables fine-tuning of BLIP-2 on Google Colab using PEFT.

---

<div class="post-metadata">

**Author:** ![dutta18](https://avatars.discourse-cdn.com/v4/letter/d/b487fb/32.png) [@dutta18](https://discuss.huggingface.co/u/dutta18)\
**Post date:** [December 22, 2023, 10:25am UTC](https://discuss.huggingface.co/t/visualbert-model-producing-runtimeerror/66360/8 "2023-12-22T10:25:19Z")

</div>

I am already using a A100 GPU. So thats not an issue. I can train them I guess. But my task specifically focuses on the discriminative settings. So far what I seen is that not many models supports this multiclass classification as you mentioned except vilt and visual-bert.

And now when I’m just trying to run the visual-bert from the [HF doc page](https://huggingface.co/docs/transformers/model_doc/visual_bert#transformers.VisualBertForQuestionAnswering). Its giving me dimensional errors. So I’m just clueless how to proceed further. I hope the code given in the doc page is tested.
