Hi, at training, I’m using the forward pass and batch_decode on the logits to get the decoded output: outputs = model( input_ids, attention_mask, dec_input_ids, dec_attention_mask, labels=dec_input_ids, ) loss, logits = outputs.loss, outputs.logi…

It may help to know the exact structure of input_ids (e.g., dimensions) and what is the context for each batch element?

input_ids is of shape (batch_size, max_len), i.e., torch.Size([16, 100]). I’m not sure what you mean by context though. This is the training routine including the output comparison: for d in data_loader: input_ids = d['enc_input_ids'].to(device) attention_mask = d['enc_attention_mask'…

In general, context refers to the input that a model has to make a prediction on (e.g., the beginning of a sentence in a next word prediction task). So, I should ask, what sort of task are you training on? Is it just span corruption prediction (i.e., the original T5 training objective)? Regarding y…

The only reason why I put generate there is for debugging purposes because I noticed that the model produced non-sense in inference mode while achieving solid accuracies on the training and validation dataset. I wanted to make sure it’s not related to saving/loading issues or encoding/tokenization i…

Ok, I do not know much about the EncoderDecoderModel so cannot comment on that. One thing I would like to understand is your setting for decoder_start_token_id. How come you decided to go with the pad token as opposed to a special token (e.g., <s> being a common one in translation). Is this how they…

The huggingface documentation of T5 (see Training) says: The PAD token is hereby used as the start-sequence token. Since decoder_start_token_id is an optional parameter for T5, I also tried it without setting it at all - but with the same result (I suspect the pad token is set as default).

Ok, so this is not the issue then. Did you also try to run the forward pass under a context manager with torch.no_grad() and got sensible outputs too?

T5 forward pass versus generate, latter outputs non-sense

Beginners

deathcrush March 25, 2021, 1:06pm 9

I happened to stumble across this post, which has a similar flavour to your error. Have a look!

Topic		Replies	Views
T5 Model Generate and Model Outputs Vastly Different Beginners	2	932	August 19, 2025
T5 weird behavior between model.forward() and model.generate 🤗Transformers	0	170	March 31, 2024
T5 - model.generate() issue Beginners	2	802	March 18, 2024
The output of T5 is not consistent on multiple sequences 🤗Transformers	1	933	May 11, 2022
T5 models have non-deterministic outputs even after disabling dropout 🤗Transformers	9	483	September 15, 2024

T5 forward pass versus generate, latter outputs non-sense

Related topics